[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:08 dist-test-slave-8xn5 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:08 dist-test-slave-8xn5 krb5kdc[395](info): set up 2 sockets
Sep 23 21:00:08 dist-test-slave-8xn5 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 21:00:09.124054 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:09 dist-test-slave-8xn5 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:09.303064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42285
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:09.633874 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:09.634231 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:09.634418 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:09.641782 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:09.641940 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:09.642053 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:09.642150 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:09.642234 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:09.652891 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:09.654290 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:09.655679 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:09.667881 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:09.667994 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:09.667845 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:09.669114 408 server_base.cc:1034] running on GCE node
I20260923 21:00:09.669920 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:09.671464 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:09.672669 408 hybrid_clock.cc:648] HybridClock initialized: now 1790197209672601 us; error 65 us; skew 500 ppm
I20260923 21:00:09.673020 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:09 dist-test-slave-8xn5 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:09.676222 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:09.677824 408 webserver.cc:460] Webserver started at http://127.0.97.62:44217/ using document root <none> and password file <none>
I20260923 21:00:09.678625 408 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:09.678752 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:09.679066 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:09.682052 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "fd0be078515345448f1e71335208f894"
format_stamp: "Formatted at 2026-09-23 21:00:09 on dist-test-slave-8xn5"
server_key: "cab96330d37954d702ef649d8432cf64"
server_key_iv: "7ecc1373ef1f74e9c1b5db80c68b5c5b"
server_key_version: "encryptionkey@0"
I20260923 21:00:09.682878 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd0be078515345448f1e71335208f894"
format_stamp: "Formatted at 2026-09-23 21:00:09 on dist-test-slave-8xn5"
server_key: "cab96330d37954d702ef649d8432cf64"
server_key_iv: "7ecc1373ef1f74e9c1b5db80c68b5c5b"
server_key_version: "encryptionkey@0"
I20260923 21:00:09.688196 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:00:09.692407 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:09.694067 408 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:00:09.694242 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "fd0be078515345448f1e71335208f894"
format_stamp: "Formatted at 2026-09-23 21:00:09 on dist-test-slave-8xn5"
server_key: "cab96330d37954d702ef649d8432cf64"
server_key_iv: "7ecc1373ef1f74e9c1b5db80c68b5c5b"
server_key_version: "encryptionkey@0"
I20260923 21:00:09.694550 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:09.713716 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:09.718528 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:09.718840 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:09.740443 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42285
I20260923 21:00:09.740605 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42285 every 8 connection(s)
I20260923 21:00:09.741885 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:09.746403 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:09.750555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 408
I20260923 21:00:09.750857 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:09.752767 388 external_mini_cluster.cc:1428] Setting key e093491af9537efd28c54eb7ae18e54e
I20260923 21:00:09.759076 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894: Bootstrap starting.
I20260923 21:00:09.762457 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:09.763635 475 log.cc:824] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:09.767815 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894: No bootstrap required, opened a new log
I20260923 21:00:09.778702 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0be078515345448f1e71335208f894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42285 } }
I20260923 21:00:09.779240 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:09.779389 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0be078515345448f1e71335208f894, State: Initialized, Role: FOLLOWER
I20260923 21:00:09.780071 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0be078515345448f1e71335208f894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42285 } }
I20260923 21:00:09.780347 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:09.780542 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:09.780728 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:09.783250 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0be078515345448f1e71335208f894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42285 } }
I20260923 21:00:09.783879 475 leader_election.cc:302] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd0be078515345448f1e71335208f894; no voters:
Sep 23 21:00:09 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:09.784628 475 leader_election.cc:288] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:09.784796 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:09.786656 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [term 1 LEADER]: Becoming Leader. State: Replica: fd0be078515345448f1e71335208f894, State: Running, Role: LEADER
I20260923 21:00:09.787314 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:09.787284 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0be078515345448f1e71335208f894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42285 } }
I20260923 21:00:09.797703 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd0be078515345448f1e71335208f894. Latest consensus state: current_term: 1 leader_uuid: "fd0be078515345448f1e71335208f894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0be078515345448f1e71335208f894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42285 } } }
I20260923 21:00:09.798192 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:09.798774 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd0be078515345448f1e71335208f894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd0be078515345448f1e71335208f894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42285 } } }
I20260923 21:00:09.799162 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:09.803884 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:09.804165 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:09.756978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.0.97.62:42285)
0923 21:00:09.758052 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:09.758070 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:09.759499 (+ 1429us) server_negotiation.cc:407] Connection header received
0923 21:00:09.769375 (+ 9876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:09.769413 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:09.770010 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:09.770592 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:09.773462 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:09.775839 (+ 2377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:09.777618 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:09.779239 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:09.779462 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:09.786612 (+ 7150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:09.786629 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:09.786661 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:09.786733 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:09.792930 (+ 6197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:09.794585 (+ 1655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:09.794600 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:09.794617 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:09.794926 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:09.795603 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:09.795610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:09.795616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:09.797756 (+ 2140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:09.801172 (+ 3416us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:09.801538 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 21:00:09.802179 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":429,"threads_started":1}
I20260923 21:00:09.810509 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:09.820742 491 catalog_manager.cc:1347] Generated new cluster ID: 5dbb69eca91a41b1b7bd52eef083b892
I20260923 21:00:09.820881 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:09.857107 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:09.859788 491 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:09.869906 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd0be078515345448f1e71335208f894: Generated new TSK 0
I20260923 21:00:09.870911 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:09.929891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42285
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:10.303755 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:10.304059 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:10.304219 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:10.312467 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:10.312599 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:10.312804 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:10.323262 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:10.324647 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:10.326184 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:10.343695 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:10.344085 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:10.344635 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:10.346087 501 server_base.cc:1034] running on GCE node
I20260923 21:00:10.346885 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:10.348274 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:10.349582 501 hybrid_clock.cc:648] HybridClock initialized: now 1790197210349497 us; error 60 us; skew 500 ppm
I20260923 21:00:10.350128 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:10 dist-test-slave-8xn5 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:10.355013 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:10.356626 501 webserver.cc:460] Webserver started at http://127.0.97.1:35895/ using document root <none> and password file <none>
I20260923 21:00:10.357335 501 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:10.357439 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:10.357712 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:10.362181 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "be79e18a08364cd39f509c99ad4d54d6"
format_stamp: "Formatted at 2026-09-23 21:00:10 on dist-test-slave-8xn5"
server_key: "136504d24428fc52dd2fe149044bec53"
server_key_iv: "da3801e8e46c3dfdd4d75095b169ab57"
server_key_version: "encryptionkey@0"
I20260923 21:00:10.363240 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "be79e18a08364cd39f509c99ad4d54d6"
format_stamp: "Formatted at 2026-09-23 21:00:10 on dist-test-slave-8xn5"
server_key: "136504d24428fc52dd2fe149044bec53"
server_key_iv: "da3801e8e46c3dfdd4d75095b169ab57"
server_key_version: "encryptionkey@0"
I20260923 21:00:10.375365 501 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.001s
I20260923 21:00:10.380343 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:10.382128 501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260923 21:00:10.382385 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "be79e18a08364cd39f509c99ad4d54d6"
format_stamp: "Formatted at 2026-09-23 21:00:10 on dist-test-slave-8xn5"
server_key: "136504d24428fc52dd2fe149044bec53"
server_key_iv: "da3801e8e46c3dfdd4d75095b169ab57"
server_key_version: "encryptionkey@0"
I20260923 21:00:10.382645 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:10.397312 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:10.402037 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:10.402411 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:10.403414 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:10.404811 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:10.404919 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:10.405071 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:10.405154 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:10.443198 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44607
I20260923 21:00:10.443216 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44607 every 8 connection(s)
I20260923 21:00:10.444635 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:10.449981 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 501
I20260923 21:00:10.450208 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:10.450866 388 external_mini_cluster.cc:1428] Setting key 394f2ef86e02d678f705cb632e61c679
Sep 23 21:00:10 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:00:10.469990 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:10.447746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49261 (local address 127.0.97.62:42285)
0923 21:00:10.448191 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:10.448196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:10.449317 (+ 1121us) server_negotiation.cc:407] Connection header received
0923 21:00:10.451657 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:10.451662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:10.451765 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:10.451880 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:10.454097 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:10.454887 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:10.456639 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:10.457089 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:10.457243 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:10.462286 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:10.462305 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:10.462314 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:10.462400 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:10.466571 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:10.467408 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:10.467418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:10.467422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:10.467604 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:10.468066 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:10.468072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:10.468076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:10.468507 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:10.468679 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:10.469429 (+ 750us) server_negotiation.cc:299] Negotiation successful
0923 21:00:10.469695 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":226,"threads_started":1}
I20260923 21:00:10.471064 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:10.448357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42285 (local address 127.0.97.1:49261)
0923 21:00:10.449137 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:10.449184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:10.451276 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:10.452242 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:10.452255 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:10.453206 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:10.453888 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:10.453915 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:10.455077 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:10.455083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:10.456461 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:10.456474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:10.457363 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:10.457371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:10.457643 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:10.458567 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:10.458607 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:10.461994 (+ 3387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:10.466764 (+ 4770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:10.466772 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:10.466789 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:10.467255 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:10.467750 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:10.467753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:10.467755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:10.467942 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:10.468653 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:10.468659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:10.469005 (+ 346us) client_negotiation.cc:769] Sending connection context
0923 21:00:10.469239 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 21:00:10.469733 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 21:00:10.472762 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42285
I20260923 21:00:10.473093 629 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:10.473845 629 heartbeater.cc:511] Master 127.0.97.62:42285 requested a full tablet report, sending...
I20260923 21:00:10.476073 440 ts_manager.cc:194] Registered new tserver with Master: be79e18a08364cd39f509c99ad4d54d6 (127.0.97.1:44607)
I20260923 21:00:10.477535 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:10.494074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42285
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:10.847451 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:10.847802 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:10.847978 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:10.855882 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:10.856055 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:10.856254 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:10.866535 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:10.867874 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:10.869253 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:10.883546 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:10.883572 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:10.883682 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:10.884801 635 server_base.cc:1034] running on GCE node
I20260923 21:00:10.885444 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:10.886615 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:10.887840 635 hybrid_clock.cc:648] HybridClock initialized: now 1790197210887777 us; error 60 us; skew 500 ppm
I20260923 21:00:10.888193 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:10 dist-test-slave-8xn5 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:10.891834 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:10.893990 635 webserver.cc:460] Webserver started at http://127.0.97.2:40215/ using document root <none> and password file <none>
I20260923 21:00:10.894843 635 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:10.894954 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:10.895334 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:10.899011 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "2402153fc16e4865b3ba37d96ecfa1e6"
format_stamp: "Formatted at 2026-09-23 21:00:10 on dist-test-slave-8xn5"
server_key: "40d81d1ba0ddf8e2e71913bab4fd98f1"
server_key_iv: "57c92c313d2650108b3f6627e478b2bb"
server_key_version: "encryptionkey@0"
I20260923 21:00:10.899864 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2402153fc16e4865b3ba37d96ecfa1e6"
format_stamp: "Formatted at 2026-09-23 21:00:10 on dist-test-slave-8xn5"
server_key: "40d81d1ba0ddf8e2e71913bab4fd98f1"
server_key_iv: "57c92c313d2650108b3f6627e478b2bb"
server_key_version: "encryptionkey@0"
I20260923 21:00:10.905793 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 21:00:10.909583 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:10.911130 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:10.911305 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "2402153fc16e4865b3ba37d96ecfa1e6"
format_stamp: "Formatted at 2026-09-23 21:00:10 on dist-test-slave-8xn5"
server_key: "40d81d1ba0ddf8e2e71913bab4fd98f1"
server_key_iv: "57c92c313d2650108b3f6627e478b2bb"
server_key_version: "encryptionkey@0"
I20260923 21:00:10.911509 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:10.944674 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:10.949687 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:10.950109 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:10.951169 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:10.952600 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:10.952673 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:10.952848 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:10.952900 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:10.990957 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39247
I20260923 21:00:10.990995 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39247 every 8 connection(s)
I20260923 21:00:10.992383 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:10.993779 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 635
I20260923 21:00:10.993979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:10.994613 388 external_mini_cluster.cc:1428] Setting key 6af237318af7d2c8cd3339909ed7b2db
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:00:11.020704 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:10.996031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57407 (local address 127.0.97.62:42285)
0923 21:00:10.996563 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:10.996570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:10.997956 (+ 1386us) server_negotiation.cc:407] Connection header received
0923 21:00:11.001776 (+ 3820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:11.001780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:11.001854 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:11.001977 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:11.004052 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.004777 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.007874 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.008354 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.008467 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.013755 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:11.013761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:11.013764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:11.013801 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:11.016728 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.017599 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.017604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.017606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.017742 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.018229 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.018257 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.018262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.018670 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:11.018835 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:11.020237 (+ 1402us) server_negotiation.cc:299] Negotiation successful
0923 21:00:11.020423 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260923 21:00:11.022485 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:10.996774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42285 (local address 127.0.97.2:57407)
0923 21:00:10.997767 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:10.997821 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:11.001388 (+ 3567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:11.002303 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:11.002362 (+ 59us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:11.003091 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:11.003837 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:11.003871 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.004951 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.004960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:11.007670 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:11.007692 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.008648 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.008659 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:11.009080 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.009938 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:11.009984 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:11.013436 (+ 3452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:11.016920 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:11.016930 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:11.016954 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:11.017421 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:11.017849 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:11.017853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:11.017855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:11.018100 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:11.018838 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:11.018845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:11.019801 (+ 956us) client_negotiation.cc:769] Sending connection context
0923 21:00:11.020154 (+ 353us) client_negotiation.cc:240] Negotiation successful
0923 21:00:11.020821 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":301,"threads_started":1}
I20260923 21:00:11.024786 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42285
I20260923 21:00:11.025087 763 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:11.025831 763 heartbeater.cc:511] Master 127.0.97.62:42285 requested a full tablet report, sending...
I20260923 21:00:11.027360 440 ts_manager.cc:194] Registered new tserver with Master: 2402153fc16e4865b3ba37d96ecfa1e6 (127.0.97.2:39247)
I20260923 21:00:11.028187 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:11.041905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42285
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:11.399645 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:11.400017 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:11.400187 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:11.408814 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:11.408996 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:11.409191 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:11.419651 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:11.421083 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:11.422664 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:11.437053 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:11.437083 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:11.437106 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:11.438807 769 server_base.cc:1034] running on GCE node
I20260923 21:00:11.439502 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:11.440513 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:11.441777 769 hybrid_clock.cc:648] HybridClock initialized: now 1790197211441699 us; error 64 us; skew 500 ppm
I20260923 21:00:11.442173 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:11.446278 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:11.449153 769 webserver.cc:460] Webserver started at http://127.0.97.3:33709/ using document root <none> and password file <none>
I20260923 21:00:11.449873 769 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:11.450022 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:11.450452 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:11.453496 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "c1d7b8de09944124a6faad80af706641"
format_stamp: "Formatted at 2026-09-23 21:00:11 on dist-test-slave-8xn5"
server_key: "2c8addf103271fa4b2515808877fa70b"
server_key_iv: "6aa3cb8f6ba72d70b466e7530e0af5ae"
server_key_version: "encryptionkey@0"
I20260923 21:00:11.454304 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c1d7b8de09944124a6faad80af706641"
format_stamp: "Formatted at 2026-09-23 21:00:11 on dist-test-slave-8xn5"
server_key: "2c8addf103271fa4b2515808877fa70b"
server_key_iv: "6aa3cb8f6ba72d70b466e7530e0af5ae"
server_key_version: "encryptionkey@0"
I20260923 21:00:11.459780 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 21:00:11.463353 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:11.464757 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:00:11.464957 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "c1d7b8de09944124a6faad80af706641"
format_stamp: "Formatted at 2026-09-23 21:00:11 on dist-test-slave-8xn5"
server_key: "2c8addf103271fa4b2515808877fa70b"
server_key_iv: "6aa3cb8f6ba72d70b466e7530e0af5ae"
server_key_version: "encryptionkey@0"
I20260923 21:00:11.465166 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:11.479110 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:11.480690 629 heartbeater.cc:503] Master 127.0.97.62:42285 was elected leader, sending a full tablet report...
I20260923 21:00:11.484450 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:11.484753 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:11.485888 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:11.487525 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:11.487629 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:11.487774 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:11.487848 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:11.526162 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43245
I20260923 21:00:11.526202 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43245 every 8 connection(s)
I20260923 21:00:11.527529 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:11.530293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 769
I20260923 21:00:11.530541 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:11.531116 388 external_mini_cluster.cc:1428] Setting key 06a0f7db290d358e987b7222ad558d21
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:11.549623 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:11.531117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54441 (local address 127.0.97.62:42285)
0923 21:00:11.531645 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:11.531653 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:11.532849 (+ 1196us) server_negotiation.cc:407] Connection header received
0923 21:00:11.534831 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:11.534836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:11.534933 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:11.535096 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:11.536700 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.537472 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.539142 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.539549 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.539687 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.543207 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:11.543212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:11.543215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:11.543248 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:11.546272 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.547395 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.547401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.547404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.547579 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.548001 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.548004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.548006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.548265 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:11.548372 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:11.549151 (+ 779us) server_negotiation.cc:299] Negotiation successful
0923 21:00:11.549337 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":236,"threads_started":1}
I20260923 21:00:11.550688 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:11.531724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42285 (local address 127.0.97.3:54441)
0923 21:00:11.532570 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:11.532613 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:11.534574 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:11.535336 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:11.535345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:11.535876 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:11.536524 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:11.536542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.537643 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.537647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:11.538991 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:11.539003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.539783 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.539790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:11.540027 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.540688 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:11.540710 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:11.542965 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:11.546489 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:11.546500 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:11.546521 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:11.547155 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:11.547694 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:11.547697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:11.547700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:11.547891 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:11.548368 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:11.548374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:11.548744 (+ 370us) client_negotiation.cc:769] Sending connection context
0923 21:00:11.549022 (+ 278us) client_negotiation.cc:240] Negotiation successful
0923 21:00:11.549440 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":335,"threads_started":1}
I20260923 21:00:11.552346 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42285
I20260923 21:00:11.552693 897 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:11.553529 897 heartbeater.cc:511] Master 127.0.97.62:42285 requested a full tablet report, sending...
I20260923 21:00:11.555173 440 ts_manager.cc:194] Registered new tserver with Master: c1d7b8de09944124a6faad80af706641 (127.0.97.3:43245)
I20260923 21:00:11.556051 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54441
I20260923 21:00:11.559947 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:11.607378 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:11.590464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.0.97.1:44607)
0923 21:00:11.591310 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:11.591317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:11.591347 (+ 30us) server_negotiation.cc:407] Connection header received
0923 21:00:11.591488 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:11.591495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:11.591709 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:11.591848 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:11.593240 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.594278 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.595771 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.596401 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.596509 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.601460 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:11.601483 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:11.601500 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:11.601547 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:11.604586 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.605317 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.605325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.605327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.605517 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.605861 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.605864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.605866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.606213 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:11.606558 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:11.606759 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 21:00:11.606968 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":586,"threads_started":1}
W20260923 21:00:11.608686 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41866
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:11.653539 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:11.638739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.0.97.1:44607)
0923 21:00:11.639000 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:11.639007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:11.639239 (+ 232us) server_negotiation.cc:407] Connection header received
0923 21:00:11.639512 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:11.639516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:11.639580 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:11.639687 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:11.641053 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.641745 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.643061 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.643495 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.643630 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.648061 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:11.648066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:11.648070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:11.648110 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:11.650934 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.651731 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.651736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.651739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.651882 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.652301 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.652305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.652307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.652593 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:11.652773 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:11.653079 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 21:00:11.653269 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 21:00:11.655195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 501
I20260923 21:00:11.668267 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:11.656271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.0.97.1:44607)
0923 21:00:11.656483 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:11.656492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:11.656785 (+ 293us) server_negotiation.cc:407] Connection header received
0923 21:00:11.656984 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:11.656988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:11.657043 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:11.657149 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:11.658371 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.659082 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.660315 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.660739 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.660885 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.663301 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:11.663306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:11.663310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:11.663340 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:11.666021 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.666727 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.666730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.666733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.666865 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.667174 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.667177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.667178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.667425 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:11.667594 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:11.667844 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 21:00:11.668002 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:00:11.669009 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:11.760489 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 501
I20260923 21:00:11.777755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 635
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:00:11.795107 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:11.778778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.0.97.2:39247)
0923 21:00:11.779286 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:11.779292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:11.779317 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:00:11.779380 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:11.779386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:11.779585 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:11.779714 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:11.780859 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.781953 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.783501 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.784123 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.784230 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.788644 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:11.788659 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:11.788676 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:11.788834 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:11.791966 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.792748 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.792756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.792803 (+ 47us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.792949 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.793386 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.793389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.793391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.793809 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:11.794029 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:11.794486 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 21:00:11.794773 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":254,"threads_started":1}
I20260923 21:00:11.796442 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:11.887248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 635
I20260923 21:00:11.910265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 769
Sep 23 21:00:11 dist-test-slave-8xn5 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:00:11.928625 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:11.911515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.97.3:43245)
0923 21:00:11.912184 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:11.912191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:11.912221 (+ 30us) server_negotiation.cc:407] Connection header received
0923 21:00:11.912284 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:11.912290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:11.912525 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:11.912705 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:11.914146 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.915354 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.916677 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:11.917241 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:11.917396 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:11.922030 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:11.922045 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:11.922061 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:11.922101 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:11.925224 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.926111 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.926119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.926122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.926566 (+ 444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:11.926957 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:11.926961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:11.926963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:11.927439 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:11.927675 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:11.928022 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 21:00:11.928241 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":356,"threads_started":1}
I20260923 21:00:11.929957 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:12.015962 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 769
I20260923 21:00:12.032562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 408
I20260923 21:00:12.033823 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:12.137449 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 408
2026-09-23T21:00:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4144 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:12 dist-test-slave-8xn5 krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:12 dist-test-slave-8xn5 krb5kdc[919](info): set up 2 sockets
Sep 23 21:00:12 dist-test-slave-8xn5 krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260923 21:00:13.238565 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:13 dist-test-slave-8xn5 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:13.422098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37677
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:13.751286 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:13.751613 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:13.751803 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:13.759234 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:13.759356 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:13.759442 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:13.759529 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:13.759611 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:13.769850 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:13.771376 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:13.772778 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:13.784945 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:13.785029 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:13.785807 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:13.785859 932 server_base.cc:1034] running on GCE node
I20260923 21:00:13.786813 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:13.788380 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:13.789629 932 hybrid_clock.cc:648] HybridClock initialized: now 1790197213789567 us; error 64 us; skew 500 ppm
I20260923 21:00:13.790000 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:13 dist-test-slave-8xn5 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:13.800460 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:13.802181 932 webserver.cc:460] Webserver started at http://127.0.97.62:39949/ using document root <none> and password file <none>
I20260923 21:00:13.803067 932 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:13.803226 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:13.803562 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:13.806819 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "5bb9fcef78f04da3b7022c53628088bd"
format_stamp: "Formatted at 2026-09-23 21:00:13 on dist-test-slave-8xn5"
server_key: "cf59798cc22475c4b445445d678e234f"
server_key_iv: "8d16789e0715fba149b47a173a995e66"
server_key_version: "encryptionkey@0"
I20260923 21:00:13.807639 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "5bb9fcef78f04da3b7022c53628088bd"
format_stamp: "Formatted at 2026-09-23 21:00:13 on dist-test-slave-8xn5"
server_key: "cf59798cc22475c4b445445d678e234f"
server_key_iv: "8d16789e0715fba149b47a173a995e66"
server_key_version: "encryptionkey@0"
I20260923 21:00:13.813279 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:00:13.817091 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:13.818535 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:13.818754 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "5bb9fcef78f04da3b7022c53628088bd"
format_stamp: "Formatted at 2026-09-23 21:00:13 on dist-test-slave-8xn5"
server_key: "cf59798cc22475c4b445445d678e234f"
server_key_iv: "8d16789e0715fba149b47a173a995e66"
server_key_version: "encryptionkey@0"
I20260923 21:00:13.819022 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:13.845229 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:13.850140 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:13.850478 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:13.869352 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37677
I20260923 21:00:13.869374 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37677 every 8 connection(s)
I20260923 21:00:13.870837 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:13.874835 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:13.880812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 932
I20260923 21:00:13.881065 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:13.881912 388 external_mini_cluster.cc:1428] Setting key e57353a6e80e5fee9e6f6e774da40965
I20260923 21:00:13.886607 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd: Bootstrap starting.
I20260923 21:00:13.890278 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:13.891561 999 log.cc:824] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:13.894572 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd: No bootstrap required, opened a new log
Sep 23 21:00:13 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:13.902053 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9fcef78f04da3b7022c53628088bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37677 } }
I20260923 21:00:13.902506 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:13.902637 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bb9fcef78f04da3b7022c53628088bd, State: Initialized, Role: FOLLOWER
I20260923 21:00:13.903326 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9fcef78f04da3b7022c53628088bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37677 } }
I20260923 21:00:13.903566 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:13.903697 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:13.903895 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:13.905289 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:13.884174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.0.97.62:37677)
0923 21:00:13.884966 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:13.884978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:13.885019 (+ 41us) server_negotiation.cc:407] Connection header received
0923 21:00:13.886789 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:13.886812 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:13.887204 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:13.887534 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:13.889308 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:13.890744 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:13.892549 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:13.893231 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:13.893424 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:13.897447 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:13.897455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:13.897470 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:13.897505 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:13.900722 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:13.901469 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:13.901476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:13.901484 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:13.901649 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:13.902035 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:13.902044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:13.902049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:13.902925 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:13.903237 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:13.903750 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 21:00:13.904168 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":367,"threads_started":1}
I20260923 21:00:13.906772 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9fcef78f04da3b7022c53628088bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37677 } }
I20260923 21:00:13.907357 999 leader_election.cc:302] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bb9fcef78f04da3b7022c53628088bd; no voters:
I20260923 21:00:13.908097 999 leader_election.cc:288] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:13.908147 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:13.910085 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [term 1 LEADER]: Becoming Leader. State: Replica: 5bb9fcef78f04da3b7022c53628088bd, State: Running, Role: LEADER
I20260923 21:00:13.910256 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:13.910611 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9fcef78f04da3b7022c53628088bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37677 } }
I20260923 21:00:13.913971 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bb9fcef78f04da3b7022c53628088bd. Latest consensus state: current_term: 1 leader_uuid: "5bb9fcef78f04da3b7022c53628088bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9fcef78f04da3b7022c53628088bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37677 } } }
I20260923 21:00:13.914428 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:13.915264 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bb9fcef78f04da3b7022c53628088bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bb9fcef78f04da3b7022c53628088bd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37677 } } }
I20260923 21:00:13.915563 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:13.916688 1015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:13.921705 1015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:13.934427 1015 catalog_manager.cc:1347] Generated new cluster ID: 7b1f6596b778474594b143d3a5d29ddc
I20260923 21:00:13.934569 1015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:13.950842 1015 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:13.952829 1015 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:13.961761 1015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bb9fcef78f04da3b7022c53628088bd: Generated new TSK 0
I20260923 21:00:13.963157 1015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:14.018174 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37677
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:14.373525 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:14.373909 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:14.374095 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:14.381614 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:14.381799 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:14.381999 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:14.392114 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37677
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:14.393504 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:14.394917 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:14.409183 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:14.409139 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:14.409214 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:14.410561 1025 server_base.cc:1034] running on GCE node
I20260923 21:00:14.411334 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:14.412413 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:14.413643 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790197214413562 us; error 81 us; skew 500 ppm
I20260923 21:00:14.414027 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:14 dist-test-slave-8xn5 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:14.417622 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:14.419283 1025 webserver.cc:460] Webserver started at http://127.0.97.1:45323/ using document root <none> and password file <none>
I20260923 21:00:14.419972 1025 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:14.420073 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:14.420322 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:14.423566 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "44360cdc72494e2ebd912a544e60c058"
format_stamp: "Formatted at 2026-09-23 21:00:14 on dist-test-slave-8xn5"
server_key: "f89b585b0cadb12a5331c509710d94ad"
server_key_iv: "792517f33ef963b9887c4ddb8d3499c1"
server_key_version: "encryptionkey@0"
I20260923 21:00:14.424448 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "44360cdc72494e2ebd912a544e60c058"
format_stamp: "Formatted at 2026-09-23 21:00:14 on dist-test-slave-8xn5"
server_key: "f89b585b0cadb12a5331c509710d94ad"
server_key_iv: "792517f33ef963b9887c4ddb8d3499c1"
server_key_version: "encryptionkey@0"
I20260923 21:00:14.429884 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:00:14.433645 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:14.435441 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:14.435601 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "44360cdc72494e2ebd912a544e60c058"
format_stamp: "Formatted at 2026-09-23 21:00:14 on dist-test-slave-8xn5"
server_key: "f89b585b0cadb12a5331c509710d94ad"
server_key_iv: "792517f33ef963b9887c4ddb8d3499c1"
server_key_version: "encryptionkey@0"
I20260923 21:00:14.435873 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:14.478130 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:14.482925 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:14.483201 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:14.484241 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:14.485704 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:14.485814 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:14.485965 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:14.486039 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:14.521482 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37233
I20260923 21:00:14.521575 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37233 every 8 connection(s)
I20260923 21:00:14.522969 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:14.527894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 1025
I20260923 21:00:14.528110 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:14.528800 388 external_mini_cluster.cc:1428] Setting key d2b1727126879b00791bef235b27be87
Sep 23 21:00:14 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:00:14.544845 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:14.525765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46969 (local address 127.0.97.62:37677)
0923 21:00:14.526194 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:14.526199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:14.527488 (+ 1289us) server_negotiation.cc:407] Connection header received
0923 21:00:14.529532 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:14.529537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:14.529648 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:14.529799 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:14.531864 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:14.532633 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:14.534233 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:14.534799 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:14.534949 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:14.538730 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:14.538736 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:14.538739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:14.538775 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:14.541600 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:14.542441 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:14.542446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:14.542448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:14.542567 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:14.543046 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:14.543053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:14.543060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:14.543330 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:14.543486 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:14.544251 (+ 765us) server_negotiation.cc:299] Negotiation successful
0923 21:00:14.544484 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":221,"threads_started":1}
I20260923 21:00:14.545970 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:14.526519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37677 (local address 127.0.97.1:46969)
0923 21:00:14.527292 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:14.527333 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:14.529192 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:14.530121 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:14.530134 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:14.531071 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:14.531684 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:14.531703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:14.532808 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:14.532812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:14.534077 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:14.534094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:14.535038 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:14.535043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:14.535346 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:14.536076 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:14.536098 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:14.538471 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:14.541767 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:14.541777 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:14.541796 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:14.542249 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:14.542692 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:14.542697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:14.542708 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:14.542922 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:14.543460 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:14.543468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:14.543843 (+ 375us) client_negotiation.cc:769] Sending connection context
0923 21:00:14.544135 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 21:00:14.544697 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":278,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 21:00:14.547955 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37677
I20260923 21:00:14.548426 1153 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:14.549234 1153 heartbeater.cc:511] Master 127.0.97.62:37677 requested a full tablet report, sending...
I20260923 21:00:14.551733 964 ts_manager.cc:194] Registered new tserver with Master: 44360cdc72494e2ebd912a544e60c058 (127.0.97.1:37233)
I20260923 21:00:14.553308 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:14.571857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37677
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:14.900715 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:14.901007 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:14.901101 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:14.908480 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:14.908613 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:14.908730 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:14.919342 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37677
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:14.920630 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:14.921957 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:14.935407 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:14.935467 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:14.935993 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:14.936908 1159 server_base.cc:1034] running on GCE node
I20260923 21:00:14.937480 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:14.938540 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:14.939738 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790197214939670 us; error 57 us; skew 500 ppm
I20260923 21:00:14.940058 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:14 dist-test-slave-8xn5 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:14.943259 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:14.944873 1159 webserver.cc:460] Webserver started at http://127.0.97.2:44009/ using document root <none> and password file <none>
I20260923 21:00:14.945596 1159 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:14.945745 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:14.946063 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:14.949188 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "7d40982df24a490dbba4ae3cb2971b84"
format_stamp: "Formatted at 2026-09-23 21:00:14 on dist-test-slave-8xn5"
server_key: "c03d2f063f083c946090377b08d7132b"
server_key_iv: "a31a06695cb2e4dd668c79fdd1503cd8"
server_key_version: "encryptionkey@0"
I20260923 21:00:14.950004 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7d40982df24a490dbba4ae3cb2971b84"
format_stamp: "Formatted at 2026-09-23 21:00:14 on dist-test-slave-8xn5"
server_key: "c03d2f063f083c946090377b08d7132b"
server_key_iv: "a31a06695cb2e4dd668c79fdd1503cd8"
server_key_version: "encryptionkey@0"
I20260923 21:00:14.955389 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260923 21:00:14.959172 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:14.960592 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:00:14.960778 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "7d40982df24a490dbba4ae3cb2971b84"
format_stamp: "Formatted at 2026-09-23 21:00:14 on dist-test-slave-8xn5"
server_key: "c03d2f063f083c946090377b08d7132b"
server_key_iv: "a31a06695cb2e4dd668c79fdd1503cd8"
server_key_version: "encryptionkey@0"
I20260923 21:00:14.960999 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:14.975925 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:14.980515 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:14.980818 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:14.981850 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:14.983438 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:14.983554 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:14.983712 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:14.983789 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:15.019158 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35381
I20260923 21:00:15.019199 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35381 every 8 connection(s)
I20260923 21:00:15.020524 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:15.029844 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 1159
I20260923 21:00:15.030041 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:15.030632 388 external_mini_cluster.cc:1428] Setting key ea17052c152216be4aba1d5122fd3901
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197214, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:15.042908 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.023199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55307 (local address 127.0.97.62:37677)
0923 21:00:15.023407 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.023411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.024703 (+ 1292us) server_negotiation.cc:407] Connection header received
0923 21:00:15.026618 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.026624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.026717 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.026837 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:15.028334 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.029033 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.030819 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.031407 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.031538 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.035995 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:15.036000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:15.036003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:15.036036 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:15.039382 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.040236 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.040245 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.040252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.040407 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.040835 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.040842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.040849 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.041234 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:15.041385 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.042136 (+ 751us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.042431 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 21:00:15.043690 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.023814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37677 (local address 127.0.97.2:55307)
0923 21:00:15.024537 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:15.024579 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:15.026268 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:15.027075 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:15.027085 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:15.027610 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:15.028163 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.028180 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.029192 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.029197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.030637 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.030650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.031681 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.031689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.031973 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.032769 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:15.032800 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:15.035729 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:15.039563 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:15.039571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:15.039586 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:15.040097 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:15.040517 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:15.040520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:15.040522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:15.040702 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:15.041362 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:15.041369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:15.041694 (+ 325us) client_negotiation.cc:769] Sending connection context
0923 21:00:15.041984 (+ 290us) client_negotiation.cc:240] Negotiation successful
0923 21:00:15.042465 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":275,"threads_started":1}
I20260923 21:00:15.045274 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37677
I20260923 21:00:15.045599 1287 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:15.046442 1287 heartbeater.cc:511] Master 127.0.97.62:37677 requested a full tablet report, sending...
I20260923 21:00:15.048019 964 ts_manager.cc:194] Registered new tserver with Master: 7d40982df24a490dbba4ae3cb2971b84 (127.0.97.2:35381)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:00:15.048965 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:15.074610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37677
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:15.405395 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:15.405673 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:15.405773 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:15.413399 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:15.413520 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:15.413635 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:15.424083 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37677
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:15.425426 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:15.427278 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:15.441347 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:15.441468 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:15.441867 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:15.442699 1292 server_base.cc:1034] running on GCE node
I20260923 21:00:15.443384 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:15.444414 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:15.445655 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790197215445601 us; error 52 us; skew 500 ppm
I20260923 21:00:15.446002 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:15.449275 1292 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:15.450903 1292 webserver.cc:460] Webserver started at http://127.0.97.3:41355/ using document root <none> and password file <none>
I20260923 21:00:15.451609 1292 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:15.451753 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:15.452073 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:15.455196 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "1f025d82eaf94737b97734ec6786bf51"
format_stamp: "Formatted at 2026-09-23 21:00:15 on dist-test-slave-8xn5"
server_key: "2d6614341c421991d42d3110b1277b2a"
server_key_iv: "2b6a1409b433060cd9e0a0bd1f3da178"
server_key_version: "encryptionkey@0"
I20260923 21:00:15.456023 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1f025d82eaf94737b97734ec6786bf51"
format_stamp: "Formatted at 2026-09-23 21:00:15 on dist-test-slave-8xn5"
server_key: "2d6614341c421991d42d3110b1277b2a"
server_key_iv: "2b6a1409b433060cd9e0a0bd1f3da178"
server_key_version: "encryptionkey@0"
I20260923 21:00:15.461602 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:00:15.465401 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:15.466954 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:15.467135 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "1f025d82eaf94737b97734ec6786bf51"
format_stamp: "Formatted at 2026-09-23 21:00:15 on dist-test-slave-8xn5"
server_key: "2d6614341c421991d42d3110b1277b2a"
server_key_iv: "2b6a1409b433060cd9e0a0bd1f3da178"
server_key_version: "encryptionkey@0"
I20260923 21:00:15.467350 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:15.488832 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:15.493418 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:15.493727 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:15.494810 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:15.496212 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:15.496294 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:15.496435 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:15.496510 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:15.531563 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43785
I20260923 21:00:15.531653 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43785 every 8 connection(s)
I20260923 21:00:15.533095 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:15.542959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 1292
I20260923 21:00:15.543166 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:15.543684 388 external_mini_cluster.cc:1428] Setting key 074c3e1e366833bbfe071b3a9b0d5100
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:15.554484 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.535972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46411 (local address 127.0.97.62:37677)
0923 21:00:15.536174 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.536179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.537495 (+ 1316us) server_negotiation.cc:407] Connection header received
0923 21:00:15.539579 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.539584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.539671 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.539800 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:15.541466 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.542258 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.544548 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.545002 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.545141 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.548796 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:15.548801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:15.548805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:15.548840 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:15.551500 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.552213 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.552217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.552219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.552427 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.552807 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.552810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.552813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.553092 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:15.553220 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.553903 (+ 683us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.554082 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 21:00:15.555567 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.536538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37677 (local address 127.0.97.3:46411)
0923 21:00:15.537295 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:15.537344 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:15.539247 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:15.540036 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:15.540044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:15.540646 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:15.541286 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.541304 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.542454 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.542458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.544400 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.544412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.545230 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.545238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.545520 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.546222 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:15.546244 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:15.548579 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:15.551642 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:15.551650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:15.551666 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:15.552107 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:15.552524 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:15.552527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:15.552529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:15.552705 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:15.553206 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:15.553212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:15.553539 (+ 327us) client_negotiation.cc:769] Sending connection context
0923 21:00:15.553791 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 21:00:15.554187 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":282,"threads_started":1}
I20260923 21:00:15.556318 1153 heartbeater.cc:503] Master 127.0.97.62:37677 was elected leader, sending a full tablet report...
I20260923 21:00:15.557348 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37677
I20260923 21:00:15.557744 1420 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:15.558804 1420 heartbeater.cc:511] Master 127.0.97.62:37677 requested a full tablet report, sending...
I20260923 21:00:15.560346 964 ts_manager.cc:194] Registered new tserver with Master: 1f025d82eaf94737b97734ec6786bf51 (127.0.97.3:43785)
I20260923 21:00:15.561160 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46411
I20260923 21:00:15.572175 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:00:15.592692 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.579312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.97.62:37677)
0923 21:00:15.579521 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.579528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.580032 (+ 504us) server_negotiation.cc:407] Connection header received
0923 21:00:15.580241 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.580257 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.580323 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.580434 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:15.581960 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.582769 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.584470 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.584879 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.585023 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.587522 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:15.587528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:15.587531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:15.587567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:15.589942 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.590702 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.590708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.590712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.590841 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.591208 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.591214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.591217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.591531 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:15.591806 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.592141 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.592376 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:00:15.600183 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:15.607789 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:15.643492 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.620948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35381 (local address 127.0.0.1:36932)
0923 21:00:15.633564 (+ 12616us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:15.633582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:15.633723 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:15.634704 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:15.634708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:15.634729 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:15.635178 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.635191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.637001 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.637006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.640115 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.640129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.642289 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.642296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.642542 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.642598 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 21:00:15.642763 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 21:00:15.642998 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12505,"mutex_wait_us":145,"thread_start_us":12353,"threads_started":1}
I20260923 21:00:15.643740 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.620945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.0.97.3:43785)
0923 21:00:15.632470 (+ 11525us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.632484 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.632541 (+ 57us) server_negotiation.cc:407] Connection header received
0923 21:00:15.633407 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.633417 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.633709 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.633915 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:15.634942 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.636783 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.640661 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.642182 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.642320 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.642572 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.643202 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.643386 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11269,"thread_start_us":11626,"threads_started":1}
I20260923 21:00:15.644088 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.620735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43785 (local address 127.0.0.1:33226)
0923 21:00:15.632166 (+ 11431us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:15.632233 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:15.632524 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:15.634238 (+ 1714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:15.634248 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:15.634282 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:15.634708 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.634719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.637001 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.637006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.640432 (+ 3426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.640456 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.642739 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.642746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.642980 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.643009 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 21:00:15.643095 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 21:00:15.643249 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11032,"mutex_wait_us":817,"thread_start_us":12033,"threads_started":1}
I20260923 21:00:15.644534 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.621063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.0.97.2:35381)
0923 21:00:15.633229 (+ 12166us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.633241 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.633857 (+ 616us) server_negotiation.cc:407] Connection header received
0923 21:00:15.633962 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.633970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.634224 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.634474 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:15.635379 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.636835 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.640270 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.641741 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.641915 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.642117 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.643992 (+ 1875us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.644186 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12007,"thread_start_us":11900,"threads_started":1}
I20260923 21:00:15.648131 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.624443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.97.1:37233)
0923 21:00:15.636773 (+ 12330us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.636785 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.637408 (+ 623us) server_negotiation.cc:407] Connection header received
0923 21:00:15.637630 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.637637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.637921 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.638125 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:15.640670 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.641951 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.644582 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.646941 (+ 2359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.647104 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.647342 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.647657 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.647810 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12112,"thread_start_us":11955,"threads_started":1}
I20260923 21:00:15.648167 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.623650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37233 (local address 127.0.0.1:38894)
0923 21:00:15.637263 (+ 13613us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:15.637282 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:15.637453 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:15.638161 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:15.638165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:15.638200 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:15.638625 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.638637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.642102 (+ 3465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.642109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.644424 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.644438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.647272 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.647280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.647459 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.647485 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 21:00:15.647594 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 21:00:15.647744 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13460,"mutex_wait_us":1,"thread_start_us":13294,"threads_started":1}
I20260923 21:00:15.648329 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 6df17406f3d148bd839a494141bbae3b (DEFAULT_TABLE table=test-table [id=a22b87c8b8ad4d7e82c358643236952a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:15.648803 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 6df17406f3d148bd839a494141bbae3b (DEFAULT_TABLE table=test-table [id=a22b87c8b8ad4d7e82c358643236952a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:15.649963 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df17406f3d148bd839a494141bbae3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:15.650079 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df17406f3d148bd839a494141bbae3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:15.653363 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 6df17406f3d148bd839a494141bbae3b (DEFAULT_TABLE table=test-table [id=a22b87c8b8ad4d7e82c358643236952a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:15.654757 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df17406f3d148bd839a494141bbae3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:15.667867 1438 tablet_bootstrap.cc:492] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84: Bootstrap starting.
I20260923 21:00:15.671805 1439 tablet_bootstrap.cc:492] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51: Bootstrap starting.
I20260923 21:00:15.673745 1438 tablet_bootstrap.cc:654] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:15.675436 1440 tablet_bootstrap.cc:492] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058: Bootstrap starting.
I20260923 21:00:15.675642 1438 log.cc:824] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:15.678627 1440 tablet_bootstrap.cc:654] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:15.679116 1438 tablet_bootstrap.cc:492] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84: No bootstrap required, opened a new log
I20260923 21:00:15.679432 1438 ts_tablet_manager.cc:1397] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260923 21:00:15.680399 1440 log.cc:824] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:15.681241 1439 tablet_bootstrap.cc:654] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:15.682546 1439 log.cc:824] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:15.684145 1440 tablet_bootstrap.cc:492] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058: No bootstrap required, opened a new log
I20260923 21:00:15.684544 1440 ts_tablet_manager.cc:1397] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260923 21:00:15.688288 1438 raft_consensus.cc:348] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.688723 1438 raft_consensus.cc:374] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:15.688869 1438 raft_consensus.cc:729] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d40982df24a490dbba4ae3cb2971b84, State: Initialized, Role: FOLLOWER
I20260923 21:00:15.689565 1438 consensus_queue.cc:260] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.689985 1439 tablet_bootstrap.cc:492] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51: No bootstrap required, opened a new log
I20260923 21:00:15.690425 1439 ts_tablet_manager.cc:1397] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260923 21:00:15.690836 1287 heartbeater.cc:503] Master 127.0.97.62:37677 was elected leader, sending a full tablet report...
I20260923 21:00:15.691406 1438 ts_tablet_manager.cc:1428] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 21:00:15.695962 1440 raft_consensus.cc:348] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.696393 1440 raft_consensus.cc:374] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:15.696529 1440 raft_consensus.cc:729] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44360cdc72494e2ebd912a544e60c058, State: Initialized, Role: FOLLOWER
I20260923 21:00:15.697264 1440 consensus_queue.cc:260] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.699533 1440 ts_tablet_manager.cc:1428] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260923 21:00:15.701429 1439 raft_consensus.cc:348] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.701977 1439 raft_consensus.cc:374] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:15.702090 1439 raft_consensus.cc:729] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f025d82eaf94737b97734ec6786bf51, State: Initialized, Role: FOLLOWER
I20260923 21:00:15.702840 1439 consensus_queue.cc:260] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.704152 1420 heartbeater.cc:503] Master 127.0.97.62:37677 was elected leader, sending a full tablet report...
I20260923 21:00:15.704828 1439 ts_tablet_manager.cc:1428] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260923 21:00:15.740804 1445 raft_consensus.cc:482] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:15.741290 1445 raft_consensus.cc:504] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.743360 1445 leader_election.cc:288] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f025d82eaf94737b97734ec6786bf51 (127.0.97.3:43785), 7d40982df24a490dbba4ae3cb2971b84 (127.0.97.2:35381)
I20260923 21:00:15.754561 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.744002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37613 (local address 127.0.97.2:35381)
0923 21:00:15.744237 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.744245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.744859 (+ 614us) server_negotiation.cc:407] Connection header received
0923 21:00:15.745064 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.745069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.745137 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.745242 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:15.746299 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.747627 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.751769 (+ 4142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.753122 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.753290 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.753401 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.754099 (+ 698us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.754251 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:00:15.754561 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.744056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59903 (local address 127.0.97.3:43785)
0923 21:00:15.744273 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.744281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.744851 (+ 570us) server_negotiation.cc:407] Connection header received
0923 21:00:15.745046 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.745052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.745134 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.745242 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:15.746366 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.747545 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.751761 (+ 4216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.753122 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.753292 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.753401 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.754099 (+ 698us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.754303 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 21:00:15.754755 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.743644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43785 (local address 127.0.97.1:59903)
0923 21:00:15.744744 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:15.744760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:15.744886 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:15.745353 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:15.745358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:15.745408 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:15.745989 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.746010 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.747787 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.747794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.751397 (+ 3603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.751426 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.753536 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.753541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.753886 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.753912 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 21:00:15.754015 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 21:00:15.754205 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":262,"spinlock_wait_cycles":17664,"thread_start_us":443,"threads_started":1}
I20260923 21:00:15.754781 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.743644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35381 (local address 127.0.97.1:37613)
0923 21:00:15.744690 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:15.744717 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:15.744932 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:15.745353 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:15.745358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:15.745424 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:15.746022 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.746032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.747788 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.747794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.751397 (+ 3603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:15.751425 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.753511 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.753524 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:15.753774 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.753886 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 21:00:15.754015 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 21:00:15.754317 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":350,"spinlock_wait_cycles":58112,"thread_start_us":787,"threads_started":1}
I20260923 21:00:15.756018 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df17406f3d148bd839a494141bbae3b" candidate_uuid: "44360cdc72494e2ebd912a544e60c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f025d82eaf94737b97734ec6786bf51" is_pre_election: true
I20260923 21:00:15.756050 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df17406f3d148bd839a494141bbae3b" candidate_uuid: "44360cdc72494e2ebd912a544e60c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40982df24a490dbba4ae3cb2971b84" is_pre_election: true
I20260923 21:00:15.756561 1375 raft_consensus.cc:2454] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44360cdc72494e2ebd912a544e60c058 in term 0.
I20260923 21:00:15.756561 1242 raft_consensus.cc:2454] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44360cdc72494e2ebd912a544e60c058 in term 0.
I20260923 21:00:15.757328 1043 leader_election.cc:302] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f025d82eaf94737b97734ec6786bf51, 44360cdc72494e2ebd912a544e60c058, 7d40982df24a490dbba4ae3cb2971b84; no voters:
I20260923 21:00:15.757776 1445 raft_consensus.cc:2781] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:15.757946 1445 raft_consensus.cc:482] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:15.758057 1445 raft_consensus.cc:3037] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:15.760486 1445 raft_consensus.cc:504] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
I20260923 21:00:15.761307 1445 leader_election.cc:288] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [CANDIDATE]: Term 1 election: Requested vote from peers 1f025d82eaf94737b97734ec6786bf51 (127.0.97.3:43785), 7d40982df24a490dbba4ae3cb2971b84 (127.0.97.2:35381)
I20260923 21:00:15.761899 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df17406f3d148bd839a494141bbae3b" candidate_uuid: "44360cdc72494e2ebd912a544e60c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f025d82eaf94737b97734ec6786bf51"
I20260923 21:00:15.761976 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df17406f3d148bd839a494141bbae3b" candidate_uuid: "44360cdc72494e2ebd912a544e60c058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d40982df24a490dbba4ae3cb2971b84"
I20260923 21:00:15.762161 1375 raft_consensus.cc:3037] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:15.762212 1242 raft_consensus.cc:3037] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:15.764750 1242 raft_consensus.cc:2454] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44360cdc72494e2ebd912a544e60c058 in term 1.
I20260923 21:00:15.765326 1375 raft_consensus.cc:2454] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44360cdc72494e2ebd912a544e60c058 in term 1.
I20260923 21:00:15.765322 1043 leader_election.cc:302] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44360cdc72494e2ebd912a544e60c058, 7d40982df24a490dbba4ae3cb2971b84; no voters:
I20260923 21:00:15.765908 1445 raft_consensus.cc:2781] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:15.766471 1445 raft_consensus.cc:686] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [term 1 LEADER]: Becoming Leader. State: Replica: 44360cdc72494e2ebd912a544e60c058, State: Running, Role: LEADER
I20260923 21:00:15.767073 1445 consensus_queue.cc:237] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } }
W20260923 21:00:15.772482 1288 tablet.cc:2369] T 6df17406f3d148bd839a494141bbae3b P 7d40982df24a490dbba4ae3cb2971b84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:00:15.772919 964 catalog_manager.cc:5495] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44360cdc72494e2ebd912a544e60c058 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "44360cdc72494e2ebd912a544e60c058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f025d82eaf94737b97734ec6786bf51" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d40982df24a490dbba4ae3cb2971b84" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44360cdc72494e2ebd912a544e60c058" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37233 } health_report { overall_health: HEALTHY } } }
W20260923 21:00:15.775490 1154 tablet.cc:2369] T 6df17406f3d148bd839a494141bbae3b P 44360cdc72494e2ebd912a544e60c058: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:00:15.784859 1421 tablet.cc:2369] T 6df17406f3d148bd839a494141bbae3b P 1f025d82eaf94737b97734ec6786bf51: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:15.820189 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.801607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.0.97.1:37233)
0923 21:00:15.801896 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.801905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.802447 (+ 542us) server_negotiation.cc:407] Connection header received
0923 21:00:15.802679 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.802685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.802784 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.802924 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:15.805057 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.805884 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.807375 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.807850 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.808012 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.813230 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:15.813248 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:15.813266 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:15.813315 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:15.816716 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.817584 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.817593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.817595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.817740 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.818223 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.818228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.818230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.818887 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:15.819223 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.819565 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.819832 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:15.882704 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.861470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.97.1:37233)
0923 21:00:15.861774 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.861782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.862193 (+ 411us) server_negotiation.cc:407] Connection header received
0923 21:00:15.862517 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.862522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.862597 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.862763 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:15.864511 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.865462 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.868023 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.868524 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.868722 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.875023 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:15.875032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:15.875037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:15.875093 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:15.879197 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.880197 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.880211 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.880213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.880436 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.880992 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.880997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.881000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.881328 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:15.881666 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.881995 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.882274 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 21:00:15 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:15.940543 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.921778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.0.97.1:37233)
0923 21:00:15.922057 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.922063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.922418 (+ 355us) server_negotiation.cc:407] Connection header received
0923 21:00:15.922733 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.922739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.922872 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.922972 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:15.924967 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.925902 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.927656 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.928168 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.928325 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.933866 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:15.933882 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:15.933888 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:15.933970 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:15.937004 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.937970 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.937977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.937980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.938157 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.938876 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.938880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.938883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.939202 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:15.939409 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.939990 (+ 581us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.940190 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260923 21:00:15.942644 1068 tablet_service.cc:2908] Rejecting scan request for tablet 6df17406f3d148bd839a494141bbae3b: Uninitialized: clean time has not yet been initialized
I20260923 21:00:15.947333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1025
I20260923 21:00:15.963053 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:15.948678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.0.97.1:37233)
0923 21:00:15.948935 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:15.948942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:15.949141 (+ 199us) server_negotiation.cc:407] Connection header received
0923 21:00:15.949445 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:15.949451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:15.949552 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:15.949675 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:15.951219 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.952028 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.953362 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:15.953804 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:15.953953 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:15.956471 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:15.956477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:15.956481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:15.956520 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:15.959866 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.960821 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.960828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.960839 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.961042 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:15.961476 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:15.961482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:15.961486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:15.961778 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:15.962046 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:15.962428 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 21:00:15.962665 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 21:00:15.964000 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:16.061340 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 1025
I20260923 21:00:16.079931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1159
Sep 23 21:00:16 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:00:16.095957 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:16.080834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.0.97.2:35381)
0923 21:00:16.081017 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:16.081022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:16.081039 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:00:16.081115 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:16.081119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:16.081205 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:16.081293 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:16.082594 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:16.083366 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:16.084885 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:16.085294 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:16.085395 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:16.089861 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:16.089876 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:16.089892 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:16.089933 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:16.093065 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:16.093805 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:16.093814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:16.093816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:16.093979 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:16.094446 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:16.094455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:16.094462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:16.094967 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:16.095195 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:16.095488 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 21:00:16.095670 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:00:16.096679 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:16.188764 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 1159
I20260923 21:00:16.207257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1292
Sep 23 21:00:16 dist-test-slave-8xn5 krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:00:16.224745 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:16.208013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.0.97.3:43785)
0923 21:00:16.208183 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:16.208187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:16.208206 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:00:16.208368 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:16.208372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:16.208443 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:16.208536 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:16.209868 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:16.210719 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:16.212931 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:16.213795 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:16.213950 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:16.218948 (+ 4998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:16.218964 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:16.218981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:16.219021 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:16.221762 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:16.222529 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:16.222537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:16.222539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:16.222672 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:16.223041 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:16.223045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:16.223047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:16.223622 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:16.223842 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:16.224077 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 21:00:16.224397 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 21:00:16.225454 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:16.314630 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 1292
I20260923 21:00:16.331763 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 932
I20260923 21:00:16.333088 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:16.450637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 932
2026-09-23T21:00:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4318 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:16 dist-test-slave-8xn5 krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:16 dist-test-slave-8xn5 krb5kdc[1468](info): set up 2 sockets
Sep 23 21:00:16 dist-test-slave-8xn5 krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260923 21:00:17.822532 388 mini_kdc.cc:121] Time spent starting KDC: real 1.323s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:17 dist-test-slave-8xn5 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:18.022893 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46529
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:18.348973 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:18.349301 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:18.349395 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:18.356668 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:18.356781 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 21:00:18.356822 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:18.356882 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:18.356931 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:18.356986 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:18.366760 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:18.368000 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:18.369270 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:18.382416 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:18.382315 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:18.382638 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:18.383471 1481 server_base.cc:1034] running on GCE node
I20260923 21:00:18.384323 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:18.385865 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:18.387310 1481 hybrid_clock.cc:648] HybridClock initialized: now 1790197218387224 us; error 76 us; skew 500 ppm
I20260923 21:00:18.387702 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:18 dist-test-slave-8xn5 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:18.391055 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:18.392481 1481 webserver.cc:460] Webserver started at http://127.0.97.62:44805/ using document root <none> and password file <none>
I20260923 21:00:18.393159 1481 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:18.393242 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:18.393512 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:18.396535 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "f50a5758d54f4e84882d1037d69b1b14"
format_stamp: "Formatted at 2026-09-23 21:00:18 on dist-test-slave-8xn5"
server_key: "23065ea7fac4423be0a390960e050c06"
server_key_iv: "3a1bb372fb06d414bc468aa1dd13341b"
server_key_version: "encryptionkey@0"
I20260923 21:00:18.397280 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "f50a5758d54f4e84882d1037d69b1b14"
format_stamp: "Formatted at 2026-09-23 21:00:18 on dist-test-slave-8xn5"
server_key: "23065ea7fac4423be0a390960e050c06"
server_key_iv: "3a1bb372fb06d414bc468aa1dd13341b"
server_key_version: "encryptionkey@0"
I20260923 21:00:18.402539 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260923 21:00:18.406549 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:18.408229 1481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:00:18.408398 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "f50a5758d54f4e84882d1037d69b1b14"
format_stamp: "Formatted at 2026-09-23 21:00:18 on dist-test-slave-8xn5"
server_key: "23065ea7fac4423be0a390960e050c06"
server_key_iv: "3a1bb372fb06d414bc468aa1dd13341b"
server_key_version: "encryptionkey@0"
I20260923 21:00:18.408612 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:18.430537 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:18.435143 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:18.435412 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:18.457180 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46529
I20260923 21:00:18.457202 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46529 every 8 connection(s)
I20260923 21:00:18.458640 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:18.460856 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 1481
I20260923 21:00:18.461093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:18.461885 388 external_mini_cluster.cc:1428] Setting key 092c748dd0ee6811ca89babc242f262c
I20260923 21:00:18.463464 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 21:00:18 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:18.478008 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14: Bootstrap starting.
I20260923 21:00:18.481443 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:18.482728 1558 log.cc:824] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:18.486267 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14: No bootstrap required, opened a new log
I20260923 21:00:18.486737 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:18.464024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.0.97.62:46529)
0923 21:00:18.464834 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:18.464846 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:18.464887 (+ 41us) server_negotiation.cc:407] Connection header received
0923 21:00:18.466479 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:18.466502 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:18.466914 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:18.467245 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:18.469088 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:18.470686 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:18.472614 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:18.473440 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:18.473617 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:18.478809 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:18.478818 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:18.478833 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:18.478877 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:18.482014 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:18.482973 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:18.482984 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:18.482995 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:18.483202 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:18.483703 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:18.483712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:18.483716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:18.484479 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:18.484766 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:18.485197 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 21:00:18.485593 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":404,"threads_started":1}
I20260923 21:00:18.493875 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50a5758d54f4e84882d1037d69b1b14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260923 21:00:18.494292 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:18.494418 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50a5758d54f4e84882d1037d69b1b14, State: Initialized, Role: FOLLOWER
I20260923 21:00:18.495019 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50a5758d54f4e84882d1037d69b1b14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260923 21:00:18.495221 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:18.495373 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:18.495555 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:18.497711 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50a5758d54f4e84882d1037d69b1b14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260923 21:00:18.498477 1558 leader_election.cc:302] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f50a5758d54f4e84882d1037d69b1b14; no voters:
I20260923 21:00:18.499099 1558 leader_election.cc:288] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:18.499370 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:18.500150 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [term 1 LEADER]: Becoming Leader. State: Replica: f50a5758d54f4e84882d1037d69b1b14, State: Running, Role: LEADER
I20260923 21:00:18.500718 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50a5758d54f4e84882d1037d69b1b14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260923 21:00:18.501375 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:18.504593 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f50a5758d54f4e84882d1037d69b1b14. Latest consensus state: current_term: 1 leader_uuid: "f50a5758d54f4e84882d1037d69b1b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50a5758d54f4e84882d1037d69b1b14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } } }
I20260923 21:00:18.505028 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:18.505887 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:18.505724 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f50a5758d54f4e84882d1037d69b1b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50a5758d54f4e84882d1037d69b1b14" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } } }
I20260923 21:00:18.506886 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:18.514002 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:18.522516 1572 catalog_manager.cc:1347] Generated new cluster ID: 13eb64c020554a7e8761a2fc8dbc157d
I20260923 21:00:18.522647 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:18.541503 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:18.543339 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:18.553282 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14: Generated new TSK 0
I20260923 21:00:18.554452 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:18.610352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46529
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:18.942281 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:18.942754 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:18.942929 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:18.950474 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:18.950641 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:18.950811 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:00:18.950965 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:18.961697 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:18.963217 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:18.964679 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:18.978315 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:18.978412 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:18.978384 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:18.979880 1584 server_base.cc:1034] running on GCE node
I20260923 21:00:18.980598 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:18.981626 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:18.982898 1584 hybrid_clock.cc:648] HybridClock initialized: now 1790197218982827 us; error 43 us; skew 500 ppm
I20260923 21:00:18.983314 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:18 dist-test-slave-8xn5 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:18.990892 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:18.992547 1584 webserver.cc:460] Webserver started at http://127.0.97.1:40279/ using document root <none> and password file <none>
I20260923 21:00:18.993341 1584 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:18.993474 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:18.993813 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:18.996901 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "20d6030a7ea94378bd98c9a0e5c5f547"
format_stamp: "Formatted at 2026-09-23 21:00:18 on dist-test-slave-8xn5"
server_key: "3e1cb2b8fd03f7324be2815cc96260ca"
server_key_iv: "4b0cedd36571c220f54c2c5d5b368461"
server_key_version: "encryptionkey@0"
I20260923 21:00:18.997826 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "20d6030a7ea94378bd98c9a0e5c5f547"
format_stamp: "Formatted at 2026-09-23 21:00:18 on dist-test-slave-8xn5"
server_key: "3e1cb2b8fd03f7324be2815cc96260ca"
server_key_iv: "4b0cedd36571c220f54c2c5d5b368461"
server_key_version: "encryptionkey@0"
I20260923 21:00:19.003486 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 21:00:19.007140 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:19.008603 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:00:19.008792 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "20d6030a7ea94378bd98c9a0e5c5f547"
format_stamp: "Formatted at 2026-09-23 21:00:18 on dist-test-slave-8xn5"
server_key: "3e1cb2b8fd03f7324be2815cc96260ca"
server_key_iv: "4b0cedd36571c220f54c2c5d5b368461"
server_key_version: "encryptionkey@0"
I20260923 21:00:19.009110 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:19.032749 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:19.037567 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:19.037855 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:19.040783 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:19.040905 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:19.041069 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:19.041150 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:00:19 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:19.066895 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.041700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58023 (local address 127.0.97.62:46529)
0923 21:00:19.042282 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:19.042291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:19.043606 (+ 1315us) server_negotiation.cc:407] Connection header received
0923 21:00:19.046518 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:19.046527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:19.046642 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:19.046733 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:19.049322 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.050487 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.053115 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.053575 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.053718 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.057966 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:19.057972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:19.057975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:19.058009 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:19.061878 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.063638 (+ 1760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.063647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.063652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.063854 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.064696 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.064701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.064705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.065003 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:19.065173 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:19.066243 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0923 21:00:19.066531 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":331,"threads_started":1}
I20260923 21:00:19.069559 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.042464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.1:58023)
0923 21:00:19.043398 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:19.043457 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:19.046113 (+ 2656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:19.047356 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:19.047367 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:19.048164 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:19.049127 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.049164 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.050712 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.050723 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.052972 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.052984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.053816 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.053822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.054053 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.054877 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:19.054925 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:19.057719 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:19.062369 (+ 4650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.062382 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.062449 (+ 67us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.063450 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.064009 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.064030 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.064036 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.064548 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.065151 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:19.065160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:19.065706 (+ 546us) client_negotiation.cc:769] Sending connection context
0923 21:00:19.067166 (+ 1460us) client_negotiation.cc:240] Negotiation successful
0923 21:00:19.067772 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":400,"threads_started":1}
I20260923 21:00:19.090205 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44361
I20260923 21:00:19.090296 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44361 every 8 connection(s)
I20260923 21:00:19.091833 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:19.094434 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46529
I20260923 21:00:19.094691 1720 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:19.095584 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.077143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.1:37149)
0923 21:00:19.078428 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:19.078449 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:19.078633 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:19.079271 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:19.079278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:19.080189 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:19.080802 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.080821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.082052 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.082057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.084276 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.084301 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.086631 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.086645 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.086895 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.087763 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:19.087791 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:19.089003 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:19.092630 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.092639 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.092646 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.093301 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.093780 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.093786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.093790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.093925 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.094538 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:19.094545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:19.094773 (+ 228us) client_negotiation.cc:769] Sending connection context
0923 21:00:19.094912 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 21:00:19.095240 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":993,"threads_started":1}
I20260923 21:00:19.095654 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.077034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37149 (local address 127.0.97.62:46529)
0923 21:00:19.077281 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:19.077285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:19.078555 (+ 1270us) server_negotiation.cc:407] Connection header received
0923 21:00:19.078809 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:19.078817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:19.078912 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:19.078993 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:19.081034 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.081831 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.084487 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.085310 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.085432 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.089313 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:19.089319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:19.089324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:19.089368 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:19.092445 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.093463 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.093470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.093473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.093657 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.094042 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.094047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.094051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.094392 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:19.094548 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:19.095059 (+ 511us) server_negotiation.cc:299] Negotiation successful
0923 21:00:19.095315 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260923 21:00:19.095680 1720 heartbeater.cc:511] Master 127.0.97.62:46529 requested a full tablet report, sending...
I20260923 21:00:19.097694 1513 ts_manager.cc:194] Registered new tserver with Master: 20d6030a7ea94378bd98c9a0e5c5f547 (127.0.97.1:44361)
I20260923 21:00:19.099633 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58023
I20260923 21:00:19.101257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 1584
I20260923 21:00:19.101482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:19.102284 388 external_mini_cluster.cc:1428] Setting key 14369892d729dd1861c8ab76e3484ae0
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:19.145942 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46529
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:19.486809 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:19.487087 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:19.487246 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:19.495358 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:19.495476 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:19.495656 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:00:19.495780 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:19.506175 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:19.507625 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:19.509043 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:19.523794 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:19.523825 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:19.523870 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:19.528064 1724 server_base.cc:1034] running on GCE node
I20260923 21:00:19.528729 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:19.529685 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:19.530908 1724 hybrid_clock.cc:648] HybridClock initialized: now 1790197219530842 us; error 39 us; skew 500 ppm
I20260923 21:00:19.531188 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:19 dist-test-slave-8xn5 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:19.534662 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:19.536183 1724 webserver.cc:460] Webserver started at http://127.0.97.2:44823/ using document root <none> and password file <none>
I20260923 21:00:19.536885 1724 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:19.536975 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:19.537276 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:19.540318 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "363d7eaa3e4d408a9a96484ce9baa4c4"
format_stamp: "Formatted at 2026-09-23 21:00:19 on dist-test-slave-8xn5"
server_key: "31a2f9d1406a657c11a497d3673ff9a8"
server_key_iv: "e19a626e30b9497bfe4730b133dd9630"
server_key_version: "encryptionkey@0"
I20260923 21:00:19.541113 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "363d7eaa3e4d408a9a96484ce9baa4c4"
format_stamp: "Formatted at 2026-09-23 21:00:19 on dist-test-slave-8xn5"
server_key: "31a2f9d1406a657c11a497d3673ff9a8"
server_key_iv: "e19a626e30b9497bfe4730b133dd9630"
server_key_version: "encryptionkey@0"
I20260923 21:00:19.546561 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 21:00:19.550012 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:19.551486 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:19.551640 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "363d7eaa3e4d408a9a96484ce9baa4c4"
format_stamp: "Formatted at 2026-09-23 21:00:19 on dist-test-slave-8xn5"
server_key: "31a2f9d1406a657c11a497d3673ff9a8"
server_key_iv: "e19a626e30b9497bfe4730b133dd9630"
server_key_version: "encryptionkey@0"
I20260923 21:00:19.551844 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:19.564680 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:19.568943 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:19.569170 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:19.571990 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:19.572155 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 21:00:19.572300 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:19.572373 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:00:19 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:19.597144 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.572889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.2:43131)
0923 21:00:19.573789 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:19.573837 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:19.575860 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:19.576746 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:19.576760 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:19.577608 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:19.578575 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.578608 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.579993 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.579998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.582292 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.582313 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.583168 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.583178 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.583632 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.584723 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:19.584764 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:19.588113 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:19.591562 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.591575 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.591602 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.592265 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.592761 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.592768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.592774 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.593134 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.593875 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:19.593883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:19.594372 (+ 489us) client_negotiation.cc:769] Sending connection context
0923 21:00:19.594784 (+ 412us) client_negotiation.cc:240] Negotiation successful
0923 21:00:19.595486 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":376,"threads_started":1}
I20260923 21:00:19.598460 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.572185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43131 (local address 127.0.97.62:46529)
0923 21:00:19.572417 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:19.572423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:19.573962 (+ 1539us) server_negotiation.cc:407] Connection header received
0923 21:00:19.576157 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:19.576163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:19.576258 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:19.576397 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:19.578771 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.579838 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.582479 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.582905 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.583052 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.588386 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:19.588391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:19.588394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:19.588428 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:19.591391 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.592430 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.592439 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.592446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.592621 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.593271 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.593277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.593281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.593719 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:19.593881 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:19.597751 (+ 3870us) server_negotiation.cc:299] Negotiation successful
0923 21:00:19.598028 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 21:00:19.619637 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35377
I20260923 21:00:19.619695 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35377 every 8 connection(s)
I20260923 21:00:19.621286 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:19.622390 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.605625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.2:47233)
0923 21:00:19.606410 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:19.606430 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:19.606578 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:19.606948 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:19.606953 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:19.607483 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:19.608025 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.608047 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.609414 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.609420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.610526 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:19.610538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.611459 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.611470 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:19.611716 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.612829 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:19.612863 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:19.614444 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:19.618939 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.618947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.618954 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.619646 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.620091 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:19.620096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:19.620099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:19.620229 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:19.621010 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:19.621015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:19.621238 (+ 223us) client_negotiation.cc:769] Sending connection context
0923 21:00:19.621385 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 21:00:19.621729 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":423,"threads_started":1}
I20260923 21:00:19.623090 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:19.605613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47233 (local address 127.0.97.62:46529)
0923 21:00:19.605813 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:19.605820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:19.606505 (+ 685us) server_negotiation.cc:407] Connection header received
0923 21:00:19.606733 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:19.606738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:19.606812 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:19.606915 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:19.608181 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.609276 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.610683 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:19.611215 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:19.611357 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:19.614701 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:19.614709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:19.614714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:19.614759 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:19.618717 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.619820 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.619826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.619830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.619972 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:19.620361 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:19.620367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:19.620372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:19.620806 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:19.620953 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:19.622582 (+ 1629us) server_negotiation.cc:299] Negotiation successful
0923 21:00:19.622835 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:00:19.623835 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46529
I20260923 21:00:19.624079 1859 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:19.624928 1859 heartbeater.cc:511] Master 127.0.97.62:46529 requested a full tablet report, sending...
I20260923 21:00:19.625846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 1724
I20260923 21:00:19.626057 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:19.626627 388 external_mini_cluster.cc:1428] Setting key 1b88d3fb6a404f563b8ebdf94d15d382
I20260923 21:00:19.627269 1512 ts_manager.cc:194] Registered new tserver with Master: 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2:35377)
I20260923 21:00:19.628731 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43131
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:19.670248 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46529
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:19.999346 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:19.999645 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:19.999743 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:20.007344 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:20.007468 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:20.007555 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:00:20.007639 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:20.018363 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:20.019654 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:20.021063 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:20.034446 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:20.034435 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:20.034801 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:20.036154 1863 server_base.cc:1034] running on GCE node
I20260923 21:00:20.037083 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:20.038121 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:20.039335 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790197220039276 us; error 60 us; skew 500 ppm
I20260923 21:00:20.039611 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:20 dist-test-slave-8xn5 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:20.042953 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:20.044503 1863 webserver.cc:460] Webserver started at http://127.0.97.3:37797/ using document root <none> and password file <none>
I20260923 21:00:20.045217 1863 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:20.045346 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:20.045674 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:20.048640 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "7e010d4da046489f9d537fe1826f4477"
format_stamp: "Formatted at 2026-09-23 21:00:20 on dist-test-slave-8xn5"
server_key: "f7d00e2e4257a3692576aecad2b53abc"
server_key_iv: "d07e14f64515809eed88e3d2316d67a2"
server_key_version: "encryptionkey@0"
I20260923 21:00:20.049552 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7e010d4da046489f9d537fe1826f4477"
format_stamp: "Formatted at 2026-09-23 21:00:20 on dist-test-slave-8xn5"
server_key: "f7d00e2e4257a3692576aecad2b53abc"
server_key_iv: "d07e14f64515809eed88e3d2316d67a2"
server_key_version: "encryptionkey@0"
I20260923 21:00:20.054948 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 21:00:20.058725 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:20.060211 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:00:20.060397 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "7e010d4da046489f9d537fe1826f4477"
format_stamp: "Formatted at 2026-09-23 21:00:20 on dist-test-slave-8xn5"
server_key: "f7d00e2e4257a3692576aecad2b53abc"
server_key_iv: "d07e14f64515809eed88e3d2316d67a2"
server_key_version: "encryptionkey@0"
I20260923 21:00:20.060652 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:20.085155 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:20.089938 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:20.090358 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:20.093986 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:20.094121 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:20.094274 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:20.094359 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:20.102579 1720 heartbeater.cc:503] Master 127.0.97.62:46529 was elected leader, sending a full tablet report...
Sep 23 21:00:20 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:20.123409 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.094008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39641 (local address 127.0.97.62:46529)
0923 21:00:20.094184 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.094191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.096084 (+ 1893us) server_negotiation.cc:407] Connection header received
0923 21:00:20.098288 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.098297 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.098433 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.098572 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:20.101270 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.102657 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.107114 (+ 4457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.107873 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.108034 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.113877 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:20.113884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:20.113889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:20.113948 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:20.118587 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.120051 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.120056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.120059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.120278 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.120882 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.120887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.120891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.121220 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:20.121409 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.122741 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.122987 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 21:00:20.125595 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.094870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.3:39641)
0923 21:00:20.095911 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.095963 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.097910 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.098828 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.098843 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.099868 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:20.101000 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.101035 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.102920 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.102931 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.106896 (+ 3965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.106919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.108330 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.108344 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.108783 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.109922 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:20.109964 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:20.113543 (+ 3579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:20.118808 (+ 5265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:20.118818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:20.118839 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:20.119822 (+ 983us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:20.120454 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:20.120458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:20.120463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:20.120738 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:20.121525 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:20.121536 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:20.122150 (+ 614us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.122594 (+ 444us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.123727 (+ 1133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":502,"threads_started":1}
I20260923 21:00:20.139487 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36119
I20260923 21:00:20.139533 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36119 every 8 connection(s)
I20260923 21:00:20.141155 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:20.143451 1998 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46529
I20260923 21:00:20.144266 1998 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:20.145105 1998 heartbeater.cc:511] Master 127.0.97.62:46529 requested a full tablet report, sending...
I20260923 21:00:20.146853 1512 ts_manager.cc:194] Registered new tserver with Master: 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119)
I20260923 21:00:20.147892 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39641
I20260923 21:00:20.148883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 1863
I20260923 21:00:20.149078 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:20.149609 388 external_mini_cluster.cc:1428] Setting key ddfa2404687d89430f5c84e0f89f1096
I20260923 21:00:20.152851 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:00:20.153111 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.137723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42693 (local address 127.0.97.62:46529)
0923 21:00:20.137924 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.137933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.138785 (+ 852us) server_negotiation.cc:407] Connection header received
0923 21:00:20.139125 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.139132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.139242 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.139338 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:20.140786 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.141646 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.143581 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.144167 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.144319 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.146632 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:20.146638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:20.146641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:20.146679 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:20.150260 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.151133 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.151138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.151141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.151293 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.151665 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.151673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.151678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.151993 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:20.152206 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.152531 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.152761 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:00:20.153210 1995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.137741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.3:42693)
0923 21:00:20.138649 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.138676 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.138927 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.139576 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.139580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.140217 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:20.140587 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.140603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.141793 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.141799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.143426 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.143451 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.144500 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.144507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.144649 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.145317 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:20.145342 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:20.146396 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:20.150439 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:20.150444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:20.150448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:20.150986 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:20.151412 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:20.151417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:20.151422 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:20.151569 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:20.152122 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:20.152126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:20.152273 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.152446 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.152764 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":513,"threads_started":1}
Sep 23 21:00:20 dist-test-slave-8xn5 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 21:00:20 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:20.203317 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.187846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.0.97.62:46529)
0923 21:00:20.188075 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.188081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.188224 (+ 143us) server_negotiation.cc:407] Connection header received
0923 21:00:20.188376 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.188380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.188451 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.188538 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:20.190434 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.191236 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.192509 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.192893 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.192984 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.197003 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:20.197009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:20.197012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:20.197053 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:20.199851 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.200693 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.200697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.200700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.200856 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.201497 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.201501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.201503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.201864 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:20.202093 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.202691 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.202892 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 21:00:20.215044 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:20.223608 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:20.247733 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.238607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36119 (local address 127.0.0.1:53418)
0923 21:00:20.239365 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.239392 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.239521 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.240722 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.240726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.240752 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:20.241181 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.241194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.243169 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.243174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.245326 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.245337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.246685 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.246691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.246876 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.246940 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.247083 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.247282 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"spinlock_wait_cycles":12160,"thread_start_us":285,"threads_started":1}
I20260923 21:00:20.247792 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.239002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.97.3:36119)
0923 21:00:20.240098 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.240107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.240138 (+ 31us) server_negotiation.cc:407] Connection header received
0923 21:00:20.240224 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.240231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.240484 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.240726 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:20.241389 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.242975 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.245474 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.246482 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.246638 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.246881 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.247243 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.247413 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":789,"threads_started":1}
I20260923 21:00:20.247923 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.237270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35377 (local address 127.0.0.1:54434)
0923 21:00:20.238153 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.238199 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.238379 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.239445 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.239454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.239510 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:20.240004 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.240023 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.242384 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.242389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.245155 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.245170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.247069 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.247076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.247274 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.247301 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.247388 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.247521 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":47,"spinlock_wait_cycles":8960,"thread_start_us":409,"threads_started":1}
I20260923 21:00:20.249166 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.237487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.0.97.2:35377)
0923 21:00:20.238237 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.238247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.238281 (+ 34us) server_negotiation.cc:407] Connection header received
0923 21:00:20.238843 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.238857 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.239160 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.240280 (+ 1120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:20.240546 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.242205 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.245342 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.246807 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.246986 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.247190 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.248635 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.248829 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":451,"threads_started":1}
I20260923 21:00:20.250982 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.239778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.0.97.1:44361)
0923 21:00:20.241382 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.241390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.241940 (+ 550us) server_negotiation.cc:407] Connection header received
0923 21:00:20.242078 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.242086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.242353 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.242586 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:20.243398 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.244829 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.248310 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.249711 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.249800 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.249953 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.250536 (+ 583us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.250697 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1441,"thread_start_us":1316,"threads_started":1}
I20260923 21:00:20.251084 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.239232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44361 (local address 127.0.0.1:58996)
0923 21:00:20.241803 (+ 2571us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.241824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.241949 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.242576 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.242586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.242613 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:20.243135 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.243149 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.245000 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.245003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.248074 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.248093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.250123 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.250128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.250310 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.250374 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.250488 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.250632 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2457,"mutex_wait_us":858,"thread_start_us":2316,"threads_started":1}
I20260923 21:00:20.252152 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 362e419caccc457581b9c2175eecf7c5 (DEFAULT_TABLE table=test-table [id=786d8d3913be46ef9a8223faeecff57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:20.252936 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 362e419caccc457581b9c2175eecf7c5 (DEFAULT_TABLE table=test-table [id=786d8d3913be46ef9a8223faeecff57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:20.253767 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362e419caccc457581b9c2175eecf7c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.254169 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362e419caccc457581b9c2175eecf7c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.254932 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 362e419caccc457581b9c2175eecf7c5 (DEFAULT_TABLE table=test-table [id=786d8d3913be46ef9a8223faeecff57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:20.256273 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 362e419caccc457581b9c2175eecf7c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.266678 2017 tablet_bootstrap.cc:492] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Bootstrap starting.
I20260923 21:00:20.271009 2017 tablet_bootstrap.cc:654] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.271127 2018 tablet_bootstrap.cc:492] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Bootstrap starting.
I20260923 21:00:20.272480 2017 log.cc:824] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:20.275097 2018 tablet_bootstrap.cc:654] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.275821 2017 tablet_bootstrap.cc:492] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: No bootstrap required, opened a new log
I20260923 21:00:20.276103 2017 ts_tablet_manager.cc:1397] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 21:00:20.276363 2018 log.cc:824] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:20.273828 2019 tablet_bootstrap.cc:492] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Bootstrap starting.
I20260923 21:00:20.281103 2019 tablet_bootstrap.cc:654] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.282382 2019 log.cc:824] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:20.284725 2018 tablet_bootstrap.cc:492] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: No bootstrap required, opened a new log
I20260923 21:00:20.285013 2018 ts_tablet_manager.cc:1397] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260923 21:00:20.285567 2019 tablet_bootstrap.cc:492] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: No bootstrap required, opened a new log
I20260923 21:00:20.285823 2019 ts_tablet_manager.cc:1397] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20260923 21:00:20.288343 2017 raft_consensus.cc:348] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.288805 2017 raft_consensus.cc:374] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.288906 2017 raft_consensus.cc:729] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363d7eaa3e4d408a9a96484ce9baa4c4, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.289644 2017 consensus_queue.cc:260] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.291225 1859 heartbeater.cc:503] Master 127.0.97.62:46529 was elected leader, sending a full tablet report...
I20260923 21:00:20.291531 2017 ts_tablet_manager.cc:1428] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260923 21:00:20.295070 2019 raft_consensus.cc:348] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.295182 2018 raft_consensus.cc:348] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.295537 2019 raft_consensus.cc:374] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.295588 2018 raft_consensus.cc:374] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.295643 2019 raft_consensus.cc:729] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d6030a7ea94378bd98c9a0e5c5f547, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.295691 2018 raft_consensus.cc:729] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e010d4da046489f9d537fe1826f4477, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.296267 2019 consensus_queue.cc:260] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.296375 2018 consensus_queue.cc:260] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.297998 2019 ts_tablet_manager.cc:1428] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260923 21:00:20.298967 1998 heartbeater.cc:503] Master 127.0.97.62:46529 was elected leader, sending a full tablet report...
I20260923 21:00:20.300537 2018 ts_tablet_manager.cc:1428] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260923 21:00:20.344746 1721 tablet.cc:2369] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:00:20.373438 1860 tablet.cc:2369] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:00:20.393216 1999 tablet.cc:2369] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:00:20.412874 2024 raft_consensus.cc:482] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:20.413239 2024 raft_consensus.cc:504] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.415520 2024 leader_election.cc:288] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2:35377), 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119)
I20260923 21:00:20.423527 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.415951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39127 (local address 127.0.97.2:35377)
0923 21:00:20.416162 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.416171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.416923 (+ 752us) server_negotiation.cc:407] Connection header received
0923 21:00:20.417264 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.417269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.417343 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.417391 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:20.418133 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.419324 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.421264 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.422476 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.422611 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.422754 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.423054 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.423216 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 21:00:20.423530 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.416091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35377 (local address 127.0.97.1:39127)
0923 21:00:20.416795 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.416822 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.417077 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.417541 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.417545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.417567 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:20.417973 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.417989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.419490 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.419495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.421108 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.421119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.422709 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.422717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.422876 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.422911 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.423013 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.423156 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":346,"threads_started":1}
I20260923 21:00:20.424896 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.416996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54273 (local address 127.0.97.3:36119)
0923 21:00:20.417229 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.417235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.417695 (+ 460us) server_negotiation.cc:407] Connection header received
0923 21:00:20.417843 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.417850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.417928 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.418038 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:20.418766 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.419776 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.422241 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.423506 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.423648 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.423747 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.424513 (+ 766us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.424663 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 21:00:20.424973 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e419caccc457581b9c2175eecf7c5" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" is_pre_election: true
I20260923 21:00:20.425557 1809 raft_consensus.cc:2454] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 0.
I20260923 21:00:20.425665 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.416894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36119 (local address 127.0.97.1:54273)
0923 21:00:20.417562 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.417583 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.417733 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.418075 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.418078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.418098 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:20.418586 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.418603 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.419914 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.419920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.422080 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.422091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.424208 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.424213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.424321 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.424342 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.425162 (+ 820us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.425346 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":305,"threads_started":1}
I20260923 21:00:20.426410 1602 leader_election.cc:302] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d6030a7ea94378bd98c9a0e5c5f547, 363d7eaa3e4d408a9a96484ce9baa4c4; no voters:
I20260923 21:00:20.426905 2024 raft_consensus.cc:2781] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:20.426746 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e419caccc457581b9c2175eecf7c5" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e010d4da046489f9d537fe1826f4477" is_pre_election: true
I20260923 21:00:20.427067 2024 raft_consensus.cc:482] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:20.427187 2024 raft_consensus.cc:3037] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:20.427301 1948 raft_consensus.cc:2454] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 0.
I20260923 21:00:20.430083 2024 raft_consensus.cc:504] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.430719 2024 leader_election.cc:288] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 election: Requested vote from peers 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2:35377), 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119)
I20260923 21:00:20.431169 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e419caccc457581b9c2175eecf7c5" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4"
I20260923 21:00:20.431344 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "362e419caccc457581b9c2175eecf7c5" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e010d4da046489f9d537fe1826f4477"
I20260923 21:00:20.431373 1809 raft_consensus.cc:3037] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:20.431532 1948 raft_consensus.cc:3037] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:20.435031 1948 raft_consensus.cc:2454] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 1.
I20260923 21:00:20.435031 1809 raft_consensus.cc:2454] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 1.
I20260923 21:00:20.435439 1602 leader_election.cc:302] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d6030a7ea94378bd98c9a0e5c5f547, 7e010d4da046489f9d537fe1826f4477; no voters:
I20260923 21:00:20.435755 2024 raft_consensus.cc:2781] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:20.436293 2024 raft_consensus.cc:686] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 LEADER]: Becoming Leader. State: Replica: 20d6030a7ea94378bd98c9a0e5c5f547, State: Running, Role: LEADER
I20260923 21:00:20.436944 2024 consensus_queue.cc:237] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } }
I20260923 21:00:20.442734 1511 catalog_manager.cc:5495] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d6030a7ea94378bd98c9a0e5c5f547 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } health_report { overall_health: HEALTHY } } }
I20260923 21:00:20.494098 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.488439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.0.97.1:44361)
0923 21:00:20.488703 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.488711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.488793 (+ 82us) server_negotiation.cc:407] Connection header received
0923 21:00:20.489045 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.489049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.489143 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.489264 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:20.489673 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.490472 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.491853 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.492283 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.492381 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.492879 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:20.493449 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:20.493590 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.493717 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.493786 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 21:00:20.502940 1948 raft_consensus.cc:1261] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Refusing update from remote peer 20d6030a7ea94378bd98c9a0e5c5f547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:20.503010 1809 raft_consensus.cc:1261] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Refusing update from remote peer 20d6030a7ea94378bd98c9a0e5c5f547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:20.503926 2028 consensus_queue.cc:1035] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:20.504341 2024 consensus_queue.cc:1035] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:20.519076 2033 mvcc.cc:205] Tried to move back new op lower bound from 7332647815169576960 to 7332647814914695168. Current Snapshot: MvccSnapshot[applied={T|T < 7332647815169576960}]
I20260923 21:00:20.537739 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:20.538029 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:20.541855 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14: Sending DeleteTablet for 3 replicas of tablet 362e419caccc457581b9c2175eecf7c5
I20260923 21:00:20.543074 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 362e419caccc457581b9c2175eecf7c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54434
I20260923 21:00:20.543294 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 362e419caccc457581b9c2175eecf7c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58996
I20260923 21:00:20.543282 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 362e419caccc457581b9c2175eecf7c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:20 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53418
I20260923 21:00:20.547519 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:20.548431 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:20.555904 2041 tablet_replica.cc:330] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: stopping tablet replica
I20260923 21:00:20.556478 2041 raft_consensus.cc:2229] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:20.557034 2041 raft_consensus.cc:2258] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:20.559131 1649 tablet_service.cc:1467] Processing CreateTablet for tablet fe4011ebf1e34c66a26ebbf30db73653 (DEFAULT_TABLE table=test-table [id=37fdf401fdc147bbab91f47d6ae35570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:20.559785 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4011ebf1e34c66a26ebbf30db73653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.562016 2043 tablet_replica.cc:330] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: stopping tablet replica
I20260923 21:00:20.562845 1927 tablet_service.cc:1467] Processing CreateTablet for tablet fe4011ebf1e34c66a26ebbf30db73653 (DEFAULT_TABLE table=test-table [id=37fdf401fdc147bbab91f47d6ae35570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:20.563357 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4011ebf1e34c66a26ebbf30db73653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.562563 2043 raft_consensus.cc:2229] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:20.564637 2043 raft_consensus.cc:2258] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:20.568589 2043 ts_tablet_manager.cc:1905] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:20.571725 1789 tablet_service.cc:1467] Processing CreateTablet for tablet fe4011ebf1e34c66a26ebbf30db73653 (DEFAULT_TABLE table=test-table [id=37fdf401fdc147bbab91f47d6ae35570]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:20.572198 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4011ebf1e34c66a26ebbf30db73653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.574283 2042 tablet_replica.cc:330] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: stopping tablet replica
I20260923 21:00:20.574717 2042 raft_consensus.cc:2229] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:20.575342 2042 raft_consensus.cc:2258] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:20.579262 2042 ts_tablet_manager.cc:1905] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:20.583289 2019 tablet_bootstrap.cc:492] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: Bootstrap starting.
I20260923 21:00:20.583767 2018 tablet_bootstrap.cc:492] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: Bootstrap starting.
I20260923 21:00:20.585042 2019 tablet_bootstrap.cc:654] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.585510 2018 tablet_bootstrap.cc:654] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.586752 2043 ts_tablet_manager.cc:1918] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:20.587018 2043 log.cc:1196] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/362e419caccc457581b9c2175eecf7c5
I20260923 21:00:20.587671 2043 ts_tablet_manager.cc:1939] T 362e419caccc457581b9c2175eecf7c5 P 7e010d4da046489f9d537fe1826f4477: Deleting consensus metadata
I20260923 21:00:20.588662 2041 ts_tablet_manager.cc:1905] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:20.589601 1499 catalog_manager.cc:4841] TS 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119): tablet 362e419caccc457581b9c2175eecf7c5 (table test-table [id=786d8d3913be46ef9a8223faeecff57d]) successfully deleted
I20260923 21:00:20.590317 2017 tablet_bootstrap.cc:492] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: Bootstrap starting.
I20260923 21:00:20.592896 2017 tablet_bootstrap.cc:654] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.595191 2019 tablet_bootstrap.cc:492] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: No bootstrap required, opened a new log
I20260923 21:00:20.595393 2019 ts_tablet_manager.cc:1397] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260923 21:00:20.596490 2019 raft_consensus.cc:348] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.596792 2019 raft_consensus.cc:374] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.596904 2019 raft_consensus.cc:729] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d6030a7ea94378bd98c9a0e5c5f547, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.596926 2041 ts_tablet_manager.cc:1918] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:20.597157 2041 log.cc:1196] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/362e419caccc457581b9c2175eecf7c5
I20260923 21:00:20.597239 2019 consensus_queue.cc:260] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.597734 2041 ts_tablet_manager.cc:1939] T 362e419caccc457581b9c2175eecf7c5 P 363d7eaa3e4d408a9a96484ce9baa4c4: Deleting consensus metadata
I20260923 21:00:20.599642 1499 catalog_manager.cc:4841] TS 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2:35377): tablet 362e419caccc457581b9c2175eecf7c5 (table test-table [id=786d8d3913be46ef9a8223faeecff57d]) successfully deleted
I20260923 21:00:20.599738 2042 ts_tablet_manager.cc:1918] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:20.600010 2042 log.cc:1196] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/362e419caccc457581b9c2175eecf7c5
I20260923 21:00:20.600749 2042 ts_tablet_manager.cc:1939] T 362e419caccc457581b9c2175eecf7c5 P 20d6030a7ea94378bd98c9a0e5c5f547: Deleting consensus metadata
I20260923 21:00:20.602504 2019 ts_tablet_manager.cc:1428] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260923 21:00:20.602625 1499 catalog_manager.cc:4841] TS 20d6030a7ea94378bd98c9a0e5c5f547 (127.0.97.1:44361): tablet 362e419caccc457581b9c2175eecf7c5 (table test-table [id=786d8d3913be46ef9a8223faeecff57d]) successfully deleted
I20260923 21:00:20.604571 2017 tablet_bootstrap.cc:492] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: No bootstrap required, opened a new log
I20260923 21:00:20.604728 2017 ts_tablet_manager.cc:1397] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260923 21:00:20.605554 2017 raft_consensus.cc:348] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.605778 2017 raft_consensus.cc:374] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.605883 2017 raft_consensus.cc:729] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363d7eaa3e4d408a9a96484ce9baa4c4, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.606011 2018 tablet_bootstrap.cc:492] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: No bootstrap required, opened a new log
I20260923 21:00:20.606160 2018 ts_tablet_manager.cc:1397] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.009s
I20260923 21:00:20.606163 2017 consensus_queue.cc:260] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.606769 2017 ts_tablet_manager.cc:1428] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 21:00:20.607481 2018 raft_consensus.cc:348] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.607674 2018 raft_consensus.cc:374] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.607785 2018 raft_consensus.cc:729] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e010d4da046489f9d537fe1826f4477, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.608080 2018 consensus_queue.cc:260] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.608633 2018 ts_tablet_manager.cc:1428] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 21:00:20.704788 2028 raft_consensus.cc:482] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:20.705075 2028 raft_consensus.cc:504] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.705847 2028 leader_election.cc:288] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2:35377), 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119)
I20260923 21:00:20.706539 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4011ebf1e34c66a26ebbf30db73653" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" is_pre_election: true
I20260923 21:00:20.706527 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4011ebf1e34c66a26ebbf30db73653" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e010d4da046489f9d537fe1826f4477" is_pre_election: true
I20260923 21:00:20.706794 1948 raft_consensus.cc:2454] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 0.
I20260923 21:00:20.706800 1809 raft_consensus.cc:2454] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 0.
I20260923 21:00:20.707213 1602 leader_election.cc:302] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d6030a7ea94378bd98c9a0e5c5f547, 7e010d4da046489f9d537fe1826f4477; no voters:
I20260923 21:00:20.707490 2028 raft_consensus.cc:2781] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:20.707604 2028 raft_consensus.cc:482] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:20.707728 2028 raft_consensus.cc:3037] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:20.709967 2028 raft_consensus.cc:504] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.710644 2028 leader_election.cc:288] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 election: Requested vote from peers 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2:35377), 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119)
I20260923 21:00:20.711167 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4011ebf1e34c66a26ebbf30db73653" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4"
I20260923 21:00:20.711215 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4011ebf1e34c66a26ebbf30db73653" candidate_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e010d4da046489f9d537fe1826f4477"
I20260923 21:00:20.711374 1809 raft_consensus.cc:3037] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:20.711424 1948 raft_consensus.cc:3037] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:20.713781 1809 raft_consensus.cc:2454] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 1.
I20260923 21:00:20.714251 1602 leader_election.cc:302] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d6030a7ea94378bd98c9a0e5c5f547, 363d7eaa3e4d408a9a96484ce9baa4c4; no voters:
I20260923 21:00:20.714377 1948 raft_consensus.cc:2454] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d6030a7ea94378bd98c9a0e5c5f547 in term 1.
I20260923 21:00:20.715204 2028 raft_consensus.cc:2781] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:20.715404 2028 raft_consensus.cc:686] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 LEADER]: Becoming Leader. State: Replica: 20d6030a7ea94378bd98c9a0e5c5f547, State: Running, Role: LEADER
I20260923 21:00:20.715672 2028 consensus_queue.cc:237] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.718490 1512 catalog_manager.cc:5495] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d6030a7ea94378bd98c9a0e5c5f547 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
Sep 23 21:00:20 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:20.771106 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.756695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.0.97.62:46529)
0923 21:00:20.757437 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:20.757443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:20.757466 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:00:20.757503 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:20.757506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:20.757555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:20.757671 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:20.758752 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.759522 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.760757 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.761159 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.761298 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.765635 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:20.765639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:20.765643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:20.765673 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:20.768368 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.769105 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.769112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.769115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.769234 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:20.769620 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:20.769625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:20.769628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:20.769884 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:20.770028 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:20.770466 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 21:00:20.770685 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":499,"threads_started":1}
I20260923 21:00:20.771196 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:20.756586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.0.1:42996)
0923 21:00:20.757095 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:20.757130 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:20.757300 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:20.757693 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:20.757697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:20.758381 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:20.758596 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.758606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.759684 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.759688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.760617 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:20.760627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:20.761317 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:20.761323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:20.761486 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:20.762192 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:20.762209 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:20.765373 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:20.768520 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:20.768537 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:20.768545 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:20.768973 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:20.769338 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:20.769344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:20.769348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:20.769492 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:20.770005 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:20.770014 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:20.770171 (+ 157us) client_negotiation.cc:769] Sending connection context
0923 21:00:20.770403 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 21:00:20.770809 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":134,"thread_start_us":261,"threads_started":1}
I20260923 21:00:20.776961 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42996:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 21:00:20.778244 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:20.786120 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 73a690c31a6749579c4a56ed3b4e72bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66d086b9ae4e4a7baf54b781f1d1791a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:00:20.786813 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a690c31a6749579c4a56ed3b4e72bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.786906 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 73a690c31a6749579c4a56ed3b4e72bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66d086b9ae4e4a7baf54b781f1d1791a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:00:20.787423 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a690c31a6749579c4a56ed3b4e72bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.793321 2019 tablet_bootstrap.cc:492] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: Bootstrap starting.
I20260923 21:00:20.794986 2017 tablet_bootstrap.cc:492] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: Bootstrap starting.
I20260923 21:00:20.795084 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 73a690c31a6749579c4a56ed3b4e72bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=66d086b9ae4e4a7baf54b781f1d1791a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:00:20.795636 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a690c31a6749579c4a56ed3b4e72bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:20.796026 2019 tablet_bootstrap.cc:654] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.796728 2017 tablet_bootstrap.cc:654] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.803166 2018 tablet_bootstrap.cc:492] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: Bootstrap starting.
I20260923 21:00:20.805369 2018 tablet_bootstrap.cc:654] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:20.808925 2017 tablet_bootstrap.cc:492] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: No bootstrap required, opened a new log
I20260923 21:00:20.809134 2017 ts_tablet_manager.cc:1397] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260923 21:00:20.810074 2017 raft_consensus.cc:348] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.810436 2017 raft_consensus.cc:374] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.810523 2018 tablet_bootstrap.cc:492] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: No bootstrap required, opened a new log
I20260923 21:00:20.810536 2017 raft_consensus.cc:729] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 363d7eaa3e4d408a9a96484ce9baa4c4, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.810705 2018 ts_tablet_manager.cc:1397] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260923 21:00:20.810915 2017 consensus_queue.cc:260] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.811447 2018 raft_consensus.cc:348] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.811455 2023 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:20.811702 2018 raft_consensus.cc:374] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.811779 2018 raft_consensus.cc:729] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e010d4da046489f9d537fe1826f4477, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.811815 2023 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:20.812041 2018 consensus_queue.cc:260] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.812466 2025 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:20.812781 2025 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:20.815485 2017 ts_tablet_manager.cc:1428] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260923 21:00:20.816041 2018 ts_tablet_manager.cc:1428] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260923 21:00:20.819669 2019 tablet_bootstrap.cc:492] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: No bootstrap required, opened a new log
I20260923 21:00:20.819880 2019 ts_tablet_manager.cc:1397] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.012s
I20260923 21:00:20.820972 2019 raft_consensus.cc:348] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.821235 2019 raft_consensus.cc:374] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:20.821336 2019 raft_consensus.cc:729] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d6030a7ea94378bd98c9a0e5c5f547, State: Initialized, Role: FOLLOWER
I20260923 21:00:20.821630 2019 consensus_queue.cc:260] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:20.822266 2019 ts_tablet_manager.cc:1428] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260923 21:00:20.822105 2028 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:20.822494 2028 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:21.153060 2023 raft_consensus.cc:482] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:21.153301 2023 raft_consensus.cc:504] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:21.154891 2023 leader_election.cc:288] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d6030a7ea94378bd98c9a0e5c5f547 (127.0.97.1:44361), 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119)
I20260923 21:00:21.163257 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.155293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44603 (local address 127.0.97.1:44361)
0923 21:00:21.156862 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.156872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.156899 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:00:21.156977 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.156984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.157064 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.157191 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:21.157773 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.158904 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.161359 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.162245 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.162436 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.162543 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.162841 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.162972 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":1300,"threads_started":1}
I20260923 21:00:21.163296 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.156244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40193 (local address 127.0.97.3:36119)
0923 21:00:21.157131 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.157137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.157154 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:00:21.157216 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.157220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.157296 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.157430 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:21.158005 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.158829 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.161504 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.162272 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.162468 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.162566 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.162864 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.162994 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":649,"threads_started":1}
I20260923 21:00:21.163417 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.155937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36119 (local address 127.0.97.2:40193)
0923 21:00:21.156605 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:21.156622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:21.156724 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:21.157483 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:21.157487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:21.157514 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:21.157843 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.157850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.158966 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.158972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.161368 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.161376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.162507 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.162513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.162710 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.162733 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 21:00:21.162841 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 21:00:21.162972 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":239,"thread_start_us":334,"threads_started":1}
I20260923 21:00:21.163923 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.155180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44361 (local address 127.0.97.2:44603)
0923 21:00:21.155826 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:21.155856 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:21.156034 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:21.157215 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:21.157220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:21.157245 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:21.157530 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.157542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.159103 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.159108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.161170 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.161185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.162458 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.162467 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.162668 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.162693 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 21:00:21.162806 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 21:00:21.162951 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":38,"thread_start_us":344,"threads_started":1}
I20260923 21:00:21.164047 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a690c31a6749579c4a56ed3b4e72bf" candidate_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e010d4da046489f9d537fe1826f4477" is_pre_election: true
I20260923 21:00:21.164323 1948 raft_consensus.cc:2454] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 363d7eaa3e4d408a9a96484ce9baa4c4 in term 0.
I20260923 21:00:21.164840 1742 leader_election.cc:302] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363d7eaa3e4d408a9a96484ce9baa4c4, 7e010d4da046489f9d537fe1826f4477; no voters:
I20260923 21:00:21.164861 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a690c31a6749579c4a56ed3b4e72bf" candidate_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" is_pre_election: true
I20260923 21:00:21.165247 1670 raft_consensus.cc:2454] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 363d7eaa3e4d408a9a96484ce9baa4c4 in term 0.
I20260923 21:00:21.165288 2023 raft_consensus.cc:2781] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:21.165423 2023 raft_consensus.cc:482] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:21.165539 2023 raft_consensus.cc:3037] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:21.167761 2023 raft_consensus.cc:504] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:21.168345 2023 leader_election.cc:288] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [CANDIDATE]: Term 1 election: Requested vote from peers 20d6030a7ea94378bd98c9a0e5c5f547 (127.0.97.1:44361), 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119)
I20260923 21:00:21.168833 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a690c31a6749579c4a56ed3b4e72bf" candidate_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d6030a7ea94378bd98c9a0e5c5f547"
I20260923 21:00:21.169070 1670 raft_consensus.cc:3037] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:21.168915 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a690c31a6749579c4a56ed3b4e72bf" candidate_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e010d4da046489f9d537fe1826f4477"
I20260923 21:00:21.169278 1948 raft_consensus.cc:3037] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:21.171463 1948 raft_consensus.cc:2454] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363d7eaa3e4d408a9a96484ce9baa4c4 in term 1.
I20260923 21:00:21.171706 1670 raft_consensus.cc:2454] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 363d7eaa3e4d408a9a96484ce9baa4c4 in term 1.
I20260923 21:00:21.171962 1742 leader_election.cc:302] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 363d7eaa3e4d408a9a96484ce9baa4c4, 7e010d4da046489f9d537fe1826f4477; no voters:
I20260923 21:00:21.172318 2023 raft_consensus.cc:2781] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:21.172765 2023 raft_consensus.cc:686] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 LEADER]: Becoming Leader. State: Replica: 363d7eaa3e4d408a9a96484ce9baa4c4, State: Running, Role: LEADER
I20260923 21:00:21.173357 2023 consensus_queue.cc:237] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } }
I20260923 21:00:21.175287 2062 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: TxnStatusTablet state changed. Reason: New leader 363d7eaa3e4d408a9a96484ce9baa4c4. Latest consensus state: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:21.175683 2062 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:00:21.176338 2064 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 21:00:21.176559 2064 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 21:00:21.177103 1513 catalog_manager.cc:5495] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } health_report { overall_health: UNKNOWN } } }
I20260923 21:00:21.215981 2028 consensus_queue.cc:1035] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:21.222692 2028 consensus_queue.cc:1035] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:21.267342 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.259079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35377 (local address 127.0.0.1:54450)
0923 21:00:21.259378 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:21.259419 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:21.259656 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:21.260334 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:21.260338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:21.260351 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:00:21.260556 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.260565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.262694 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.262699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.264074 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.264090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.265306 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.265311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.265524 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.265594 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:00:21.266831 (+ 1237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:00:21.266861 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 21:00:21.266944 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 21:00:21.267079 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 21:00:21.267457 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.259189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.0.97.2:35377)
0923 21:00:21.259869 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.259876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.259902 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:00:21.260003 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.260008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.260114 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.260290 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:21.260725 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.262494 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.264232 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.265032 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.265242 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.265845 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:21.266708 (+ 863us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:21.266849 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.266989 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.267081 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":406,"threads_started":1}
I20260923 21:00:21.272667 1948 raft_consensus.cc:1261] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Refusing update from remote peer 363d7eaa3e4d408a9a96484ce9baa4c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:21.272836 1670 raft_consensus.cc:1261] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Refusing update from remote peer 363d7eaa3e4d408a9a96484ce9baa4c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:21.273514 2062 consensus_queue.cc:1035] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:21.273864 2023 consensus_queue.cc:1035] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:21.276952 2025 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: TxnStatusTablet state changed. Reason: New leader 363d7eaa3e4d408a9a96484ce9baa4c4. Latest consensus state: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:21.277249 2025 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:21.280018 2072 mvcc.cc:205] Tried to move back new op lower bound from 7332647818324381696 to 7332647817929822208. Current Snapshot: MvccSnapshot[applied={T|T < 7332647818324381696}]
I20260923 21:00:21.288913 2069 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: TxnStatusTablet state changed. Reason: New leader 363d7eaa3e4d408a9a96484ce9baa4c4. Latest consensus state: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:21.289382 2069 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:21.289841 2062 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:21.290134 2062 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:00:21.290391 2025 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:21.290618 2025 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 7e010d4da046489f9d537fe1826f4477: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:21.293852 2062 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:21.294126 2062 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 363d7eaa3e4d408a9a96484ce9baa4c4: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:00:21.301946 2069 tablet_replica.cc:439] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d6030a7ea94378bd98c9a0e5c5f547" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44361 } } peers { permanent_uuid: "363d7eaa3e4d408a9a96484ce9baa4c4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35377 } } peers { permanent_uuid: "7e010d4da046489f9d537fe1826f4477" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36119 } } }
I20260923 21:00:21.302228 2069 tablet_replica.cc:442] T 73a690c31a6749579c4a56ed3b4e72bf P 20d6030a7ea94378bd98c9a0e5c5f547: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:21.320058 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.312772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35377 (local address 127.0.97.1:52607)
0923 21:00:21.314818 (+ 2046us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:21.314835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:21.315020 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:21.315374 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:21.315378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:21.315390 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:00:21.315679 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.315688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.316789 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.316794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.317869 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.317882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.318894 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.318900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.319025 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.319087 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:00:21.319595 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:00:21.319603 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 21:00:21.319671 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 21:00:21.319780 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1886,"thread_start_us":1725,"threads_started":1}
I20260923 21:00:21.320111 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.313208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52607 (local address 127.0.97.2:35377)
0923 21:00:21.313390 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.313395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.314944 (+ 1549us) server_negotiation.cc:407] Connection header received
0923 21:00:21.315170 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.315176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.315253 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.315359 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:21.315826 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.316638 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.318042 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.318681 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.318825 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.319308 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:21.319462 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:21.319578 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.319723 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.319827 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 21:00:21.361965 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.355337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35925 (local address 127.0.97.1:44361)
0923 21:00:21.355553 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.355563 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.356328 (+ 765us) server_negotiation.cc:407] Connection header received
0923 21:00:21.356559 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.356565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.356650 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.356747 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:21.357360 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.358196 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.359966 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.360409 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.360569 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.361153 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:21.361288 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:21.361386 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.361600 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.361686 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:00:21.362154 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.355391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44361 (local address 127.0.97.2:35925)
0923 21:00:21.356213 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:21.356234 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:21.356396 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:21.356785 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:21.356792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:21.356805 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:00:21.357191 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.357201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.358368 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.358377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.359803 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:21.359816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.360591 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.360597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:21.360858 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.360925 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:00:21.361433 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:00:21.361456 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 21:00:21.361565 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 21:00:21.361708 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":557,"threads_started":1}
I20260923 21:00:21.412765 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:21.413003 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:21.416723 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f50a5758d54f4e84882d1037d69b1b14: Sending DeleteTablet for 3 replicas of tablet fe4011ebf1e34c66a26ebbf30db73653
I20260923 21:00:21.417693 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4011ebf1e34c66a26ebbf30db73653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58996
I20260923 21:00:21.418001 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4011ebf1e34c66a26ebbf30db73653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53418
I20260923 21:00:21.421427 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4011ebf1e34c66a26ebbf30db73653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54434
I20260923 21:00:21.433656 2088 tablet_replica.cc:330] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: stopping tablet replica
I20260923 21:00:21.433979 2088 raft_consensus.cc:2229] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:21.434631 2088 raft_consensus.cc:2258] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:21.436174 2088 ts_tablet_manager.cc:1905] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:21.436859 2091 tablet_replica.cc:330] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: stopping tablet replica
I20260923 21:00:21.437345 2089 tablet_replica.cc:330] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: stopping tablet replica
I20260923 21:00:21.437800 2089 raft_consensus.cc:2229] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:21.438100 2091 raft_consensus.cc:2229] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:21.438501 2089 raft_consensus.cc:2258] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:21.438796 2091 raft_consensus.cc:2258] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:21.440191 2089 ts_tablet_manager.cc:1905] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:21.441241 2091 ts_tablet_manager.cc:1905] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:21.447146 2088 ts_tablet_manager.cc:1918] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:21.447479 2088 log.cc:1196] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/fe4011ebf1e34c66a26ebbf30db73653
I20260923 21:00:21.448097 2088 ts_tablet_manager.cc:1939] T fe4011ebf1e34c66a26ebbf30db73653 P 20d6030a7ea94378bd98c9a0e5c5f547: Deleting consensus metadata
I20260923 21:00:21.449654 1499 catalog_manager.cc:4841] TS 20d6030a7ea94378bd98c9a0e5c5f547 (127.0.97.1:44361): tablet fe4011ebf1e34c66a26ebbf30db73653 (table test-table [id=37fdf401fdc147bbab91f47d6ae35570]) successfully deleted
I20260923 21:00:21.453629 2089 ts_tablet_manager.cc:1918] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:21.453794 2089 log.cc:1196] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/fe4011ebf1e34c66a26ebbf30db73653
I20260923 21:00:21.454272 2089 ts_tablet_manager.cc:1939] T fe4011ebf1e34c66a26ebbf30db73653 P 7e010d4da046489f9d537fe1826f4477: Deleting consensus metadata
I20260923 21:00:21.455487 1499 catalog_manager.cc:4841] TS 7e010d4da046489f9d537fe1826f4477 (127.0.97.3:36119): tablet fe4011ebf1e34c66a26ebbf30db73653 (table test-table [id=37fdf401fdc147bbab91f47d6ae35570]) successfully deleted
I20260923 21:00:21.458905 2091 ts_tablet_manager.cc:1918] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:21.459072 2091 log.cc:1196] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/fe4011ebf1e34c66a26ebbf30db73653
I20260923 21:00:21.459563 2091 ts_tablet_manager.cc:1939] T fe4011ebf1e34c66a26ebbf30db73653 P 363d7eaa3e4d408a9a96484ce9baa4c4: Deleting consensus metadata
Sep 23 21:00:21 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:21.460728 1499 catalog_manager.cc:4841] TS 363d7eaa3e4d408a9a96484ce9baa4c4 (127.0.97.2:35377): tablet fe4011ebf1e34c66a26ebbf30db73653 (table test-table [id=37fdf401fdc147bbab91f47d6ae35570]) successfully deleted
I20260923 21:00:21.467725 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.427332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.0.97.1:44361)
0923 21:00:21.427598 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.427607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.450683 (+ 23076us) server_negotiation.cc:407] Connection header received
0923 21:00:21.450900 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.450906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.450986 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.451079 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:21.453103 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.454267 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.456120 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.456602 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.456742 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.461522 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:21.461537 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:21.461552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:21.461587 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:21.464691 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.465586 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.465600 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.465603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.465733 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.466106 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.466115 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.466117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.466532 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:21.466866 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.467126 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.467329 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260923 21:00:21.468560 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59022
I20260923 21:00:21.499035 2095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.472000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.97.62:46529)
0923 21:00:21.484867 (+ 12867us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.484878 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.485453 (+ 575us) server_negotiation.cc:407] Connection header received
0923 21:00:21.485715 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.485720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.485808 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.485936 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:21.487434 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.488290 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.489558 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.490506 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.490659 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.493240 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:21.493251 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:21.493254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:21.493291 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:21.496139 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.496885 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.496891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.496896 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.497044 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.497449 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.497456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.497460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.497795 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:21.498054 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.498429 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.498663 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12579,"thread_start_us":12350,"threads_started":1}
W20260923 21:00:21.500172 1513 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43010
I20260923 21:00:21.502445 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1584
I20260923 21:00:21.519733 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.504024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.97.1:44361)
0923 21:00:21.504228 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.504237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.504809 (+ 572us) server_negotiation.cc:407] Connection header received
0923 21:00:21.505101 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.505106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.505185 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.505292 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:21.506645 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.507357 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.508571 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.509015 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.509160 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.511771 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:21.511776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:21.511779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:21.511817 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:21.516374 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.517463 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.517471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.517474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.517644 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.518133 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.518138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.518141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.518555 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:21.518762 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.519199 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.519414 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 21:00:21.520421 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59028
W20260923 21:00:21.521003 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:21.521214 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 1584
I20260923 21:00:21.544754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1724
Sep 23 21:00:21 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:00:21.563273 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.545856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.0.97.2:35377)
0923 21:00:21.546069 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.546073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.546090 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:00:21.546146 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.546150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.546249 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.546404 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:21.548003 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.548796 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.550762 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.551258 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.551384 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.556686 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:21.556701 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:21.556717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:21.556765 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:21.560278 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.561057 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.561065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.561067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.561216 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.561617 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.561620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.561622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.561998 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:21.562322 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.562684 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.562921 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 21:00:21.563964 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54462
W20260923 21:00:21.564813 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:21.565009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 1724
I20260923 21:00:21.585484 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1863
Sep 23 21:00:21 dist-test-slave-8xn5 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:00:21.602720 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:21.586494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.0.97.3:36119)
0923 21:00:21.586674 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:21.586679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:21.586697 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:00:21.586855 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:21.586859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:21.587029 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:21.587120 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:21.588430 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.589120 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.590630 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:21.591207 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:21.591319 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:21.596017 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:21.596036 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:21.596053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:21.596097 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:21.599503 (+ 3406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.600294 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.600303 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.600305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.600458 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:21.600845 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:21.600849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:21.600851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:21.601375 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:21.601606 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:21.602119 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 21:00:21.602326 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 21:00:21.603446 1992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53424
W20260923 21:00:21.604184 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:21.604364 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 1863
I20260923 21:00:21.621155 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 1481
I20260923 21:00:21.622470 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:21.739850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 1481
2026-09-23T21:00:21Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5286 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:21 dist-test-slave-8xn5 krb5kdc[2100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:21 dist-test-slave-8xn5 krb5kdc[2100](info): set up 2 sockets
Sep 23 21:00:21 dist-test-slave-8xn5 krb5kdc[2100](info): commencing operation
krb5kdc: starting...
W20260923 21:00:22.874117 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260923 21:00:23.033751 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.159s user 0.003s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260923 21:00:23.223038 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.189s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:23 dist-test-slave-8xn5 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:23.371807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37889
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:23.713660 2113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:23.713976 2113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:23.714140 2113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:23.722210 2113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:23.722424 2113 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 21:00:23.722506 2113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:23.722659 2113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:23.722736 2113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:23.722829 2113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:23.733290 2113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37889
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:23.734694 2113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:23.736137 2113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:23.749150 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:23.749317 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:23.749150 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:23.750303 2113 server_base.cc:1034] running on GCE node
I20260923 21:00:23.751251 2113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:23.752802 2113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:23.754102 2113 hybrid_clock.cc:648] HybridClock initialized: now 1790197223754020 us; error 55 us; skew 500 ppm
I20260923 21:00:23.754547 2113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:23 dist-test-slave-8xn5 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:23.757989 2113 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:23.759783 2113 webserver.cc:460] Webserver started at http://127.0.97.62:43517/ using document root <none> and password file <none>
I20260923 21:00:23.760556 2113 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:23.760650 2113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:23.761070 2113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:23.764722 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "e442478b33ca45bab91a7fbaf062d1ad"
format_stamp: "Formatted at 2026-09-23 21:00:23 on dist-test-slave-8xn5"
server_key: "0c602f0c0a6201e0551a927d36eebc30"
server_key_iv: "25008b24b46c2df45f32fa50c843a6a8"
server_key_version: "encryptionkey@0"
I20260923 21:00:23.765683 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "e442478b33ca45bab91a7fbaf062d1ad"
format_stamp: "Formatted at 2026-09-23 21:00:23 on dist-test-slave-8xn5"
server_key: "0c602f0c0a6201e0551a927d36eebc30"
server_key_iv: "25008b24b46c2df45f32fa50c843a6a8"
server_key_version: "encryptionkey@0"
I20260923 21:00:23.771903 2113 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260923 21:00:23.776410 2128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:23.778195 2113 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 21:00:23.778574 2113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "e442478b33ca45bab91a7fbaf062d1ad"
format_stamp: "Formatted at 2026-09-23 21:00:23 on dist-test-slave-8xn5"
server_key: "0c602f0c0a6201e0551a927d36eebc30"
server_key_iv: "25008b24b46c2df45f32fa50c843a6a8"
server_key_version: "encryptionkey@0"
I20260923 21:00:23.778785 2113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:23.792850 2113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:23.797487 2113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:23.797726 2113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:23.819892 2113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37889
I20260923 21:00:23.819912 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37889 every 8 connection(s)
I20260923 21:00:23.821501 2113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:23.825852 2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:23.831423 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 2113
I20260923 21:00:23.831687 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:23.832347 388 external_mini_cluster.cc:1428] Setting key 264a052620482bca7f30b8571cc4961a
I20260923 21:00:23.837237 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Bootstrap starting.
I20260923 21:00:23.841200 2190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:23.842649 2190 log.cc:824] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:23.846380 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: No bootstrap required, opened a new log
Sep 23 21:00:23 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:23.854779 2190 raft_consensus.cc:348] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e442478b33ca45bab91a7fbaf062d1ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37889 } }
I20260923 21:00:23.855316 2190 raft_consensus.cc:374] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:23.855432 2190 raft_consensus.cc:729] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e442478b33ca45bab91a7fbaf062d1ad, State: Initialized, Role: FOLLOWER
I20260923 21:00:23.856154 2190 consensus_queue.cc:260] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e442478b33ca45bab91a7fbaf062d1ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37889 } }
I20260923 21:00:23.856498 2190 raft_consensus.cc:388] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:23.856659 2190 raft_consensus.cc:482] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:23.856855 2190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:23.857806 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:23.834465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.97.62:37889)
0923 21:00:23.835343 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:23.835355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:23.835399 (+ 44us) server_negotiation.cc:407] Connection header received
0923 21:00:23.837135 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:23.837159 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:23.837626 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:23.837983 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:23.839955 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:23.841442 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:23.843287 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:23.844424 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:23.844584 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:23.849256 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:23.849268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:23.849286 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:23.849324 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:23.852659 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:23.853504 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:23.853512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:23.853520 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:23.853706 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:23.854187 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:23.854192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:23.854195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:23.855109 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:23.855434 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:23.856039 (+ 605us) server_negotiation.cc:299] Negotiation successful
0923 21:00:23.856531 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":420,"threads_started":1}
I20260923 21:00:23.859447 2190 raft_consensus.cc:504] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e442478b33ca45bab91a7fbaf062d1ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37889 } }
I20260923 21:00:23.859968 2190 leader_election.cc:302] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e442478b33ca45bab91a7fbaf062d1ad; no voters:
I20260923 21:00:23.860589 2190 leader_election.cc:288] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:23.860801 2195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:23.861636 2195 raft_consensus.cc:686] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [term 1 LEADER]: Becoming Leader. State: Replica: e442478b33ca45bab91a7fbaf062d1ad, State: Running, Role: LEADER
I20260923 21:00:23.862175 2195 consensus_queue.cc:237] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e442478b33ca45bab91a7fbaf062d1ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37889 } }
I20260923 21:00:23.863018 2190 sys_catalog.cc:564] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:23.864845 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e442478b33ca45bab91a7fbaf062d1ad. Latest consensus state: current_term: 1 leader_uuid: "e442478b33ca45bab91a7fbaf062d1ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e442478b33ca45bab91a7fbaf062d1ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37889 } } }
I20260923 21:00:23.865181 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:23.865893 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e442478b33ca45bab91a7fbaf062d1ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e442478b33ca45bab91a7fbaf062d1ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37889 } } }
I20260923 21:00:23.866086 2201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:23.866175 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:23.872814 2201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:23.882380 2201 catalog_manager.cc:1347] Generated new cluster ID: b2f1ff3003e1408a926932c52eabf807
I20260923 21:00:23.882522 2201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:23.896656 2201 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:23.898728 2201 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:23.912567 2201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Generated new TSK 0
I20260923 21:00:23.913776 2201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:23.963476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37889
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:24.316264 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:24.316676 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:24.316861 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:24.324595 2216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:24.324764 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:24.324931 2216 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 21:00:24.325022 2216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:00:24.325166 2216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:24.335415 2216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:24.336866 2216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:24.338315 2216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:24.352391 2221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:24.352428 2222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:24.352468 2224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:24.353940 2216 server_base.cc:1034] running on GCE node
I20260923 21:00:24.354677 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:24.355764 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:24.357023 2216 hybrid_clock.cc:648] HybridClock initialized: now 1790197224356958 us; error 50 us; skew 500 ppm
I20260923 21:00:24.357358 2216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:24 dist-test-slave-8xn5 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:24.367043 2216 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:24.368640 2216 webserver.cc:460] Webserver started at http://127.0.97.1:37139/ using document root <none> and password file <none>
I20260923 21:00:24.369418 2216 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:24.369556 2216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:24.369889 2216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:24.373020 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d"
format_stamp: "Formatted at 2026-09-23 21:00:24 on dist-test-slave-8xn5"
server_key: "8cdbc6580c78c20785eba7cac3b0e8be"
server_key_iv: "0f460f34f9cfa3c80c071042491d5b9d"
server_key_version: "encryptionkey@0"
I20260923 21:00:24.373762 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d"
format_stamp: "Formatted at 2026-09-23 21:00:24 on dist-test-slave-8xn5"
server_key: "8cdbc6580c78c20785eba7cac3b0e8be"
server_key_iv: "0f460f34f9cfa3c80c071042491d5b9d"
server_key_version: "encryptionkey@0"
I20260923 21:00:24.379544 2216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 21:00:24.383327 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:24.384915 2216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:00:24.385072 2216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d"
format_stamp: "Formatted at 2026-09-23 21:00:24 on dist-test-slave-8xn5"
server_key: "8cdbc6580c78c20785eba7cac3b0e8be"
server_key_iv: "0f460f34f9cfa3c80c071042491d5b9d"
server_key_version: "encryptionkey@0"
I20260923 21:00:24.385308 2216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:24.405165 2216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:24.409469 2216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:24.409786 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:24.413002 2216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:24.413123 2216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:24.413291 2216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:24.413406 2216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:00:24 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:24.437608 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.413584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54275 (local address 127.0.97.62:37889)
0923 21:00:24.414085 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:24.414091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:24.415416 (+ 1325us) server_negotiation.cc:407] Connection header received
0923 21:00:24.417642 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:24.417649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:24.417752 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:24.417929 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:24.420678 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.422010 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.423872 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.424675 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.424926 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.430867 (+ 5941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:24.430873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:24.430876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:24.430912 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:24.434312 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.435184 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.435189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.435193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.435315 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.435783 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.435788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.435791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.436075 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:24.436234 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:24.437034 (+ 800us) server_negotiation.cc:299] Negotiation successful
0923 21:00:24.437252 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
I20260923 21:00:24.438959 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.414276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37889 (local address 127.0.97.1:54275)
0923 21:00:24.415221 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:24.415278 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:24.417321 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:24.418262 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:24.418274 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:24.419184 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:24.420336 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.420363 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.422242 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.422248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.423705 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.423718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.425429 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.425440 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.425760 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.426785 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:24.426830 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:24.430516 (+ 3686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:24.434542 (+ 4026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.434551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.434577 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.435044 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.435436 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.435441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.435445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.435657 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.436215 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:24.436223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:24.436660 (+ 437us) client_negotiation.cc:769] Sending connection context
0923 21:00:24.436930 (+ 270us) client_negotiation.cc:240] Negotiation successful
0923 21:00:24.437508 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":382,"threads_started":1}
I20260923 21:00:24.459662 2216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37979
I20260923 21:00:24.459684 2351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37979 every 8 connection(s)
I20260923 21:00:24.461474 2216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:24.464191 2352 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37889
I20260923 21:00:24.464434 2352 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:24.465312 2352 heartbeater.cc:511] Master 127.0.97.62:37889 requested a full tablet report, sending...
I20260923 21:00:24.465788 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.448044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37889 (local address 127.0.97.1:36555)
0923 21:00:24.449154 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:24.449171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:24.449334 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:24.449893 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:24.449903 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:24.450784 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:24.451184 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.451194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.452529 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.452537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.454990 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.455012 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.456358 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.456372 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.456622 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.457566 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:24.457603 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:24.458581 (+ 978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:24.462703 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.462709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.462713 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.463454 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.463913 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.463917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.463921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.464080 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.464751 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:24.464755 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:24.464921 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 21:00:24.465112 (+ 191us) client_negotiation.cc:240] Negotiation successful
0923 21:00:24.465366 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":535,"threads_started":1}
I20260923 21:00:24.465850 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.447765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36555 (local address 127.0.97.62:37889)
0923 21:00:24.447980 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:24.447986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:24.449263 (+ 1277us) server_negotiation.cc:407] Connection header received
0923 21:00:24.449476 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:24.449483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:24.449577 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:24.449687 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:24.451342 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.452352 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.455220 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.456056 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.456219 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.458841 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:24.458853 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:24.458861 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:24.458915 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:24.462475 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.463615 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.463620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.463625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.463790 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.464227 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.464232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.464235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.464635 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:24.464786 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:24.465263 (+ 477us) server_negotiation.cc:299] Negotiation successful
0923 21:00:24.465514 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 21:00:24.466334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 2216
I20260923 21:00:24.466548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:24.467273 388 external_mini_cluster.cc:1428] Setting key a6f1ec722652e82dafc18de0e99ac294
I20260923 21:00:24.468024 2144 ts_manager.cc:194] Registered new tserver with Master: e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979)
I20260923 21:00:24.469987 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54275
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:24.514464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37889
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:24.850878 2356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:24.851227 2356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:24.851419 2356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:24.859118 2356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:24.859298 2356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:24.859467 2356 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 21:00:24.859558 2356 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:00:24.859674 2356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:24.870784 2356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:24.872233 2356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:24.873646 2356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:24.888141 2364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:24.888095 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:24.888175 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:24.888696 2356 server_base.cc:1034] running on GCE node
I20260923 21:00:24.889449 2356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:24.890597 2356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:24.891901 2356 hybrid_clock.cc:648] HybridClock initialized: now 1790197224891849 us; error 43 us; skew 500 ppm
I20260923 21:00:24.892346 2356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:24 dist-test-slave-8xn5 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:24.895833 2356 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:24.897502 2356 webserver.cc:460] Webserver started at http://127.0.97.2:37273/ using document root <none> and password file <none>
I20260923 21:00:24.898229 2356 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:24.898417 2356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:24.898746 2356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:24.901791 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
format_stamp: "Formatted at 2026-09-23 21:00:24 on dist-test-slave-8xn5"
server_key: "394d8061392710334340349a081540b0"
server_key_iv: "03b3fb1c7083309bed883cfcf7905f48"
server_key_version: "encryptionkey@0"
I20260923 21:00:24.902667 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
format_stamp: "Formatted at 2026-09-23 21:00:24 on dist-test-slave-8xn5"
server_key: "394d8061392710334340349a081540b0"
server_key_iv: "03b3fb1c7083309bed883cfcf7905f48"
server_key_version: "encryptionkey@0"
I20260923 21:00:24.908107 2356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:00:24.911863 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:24.913542 2356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:00:24.913736 2356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
format_stamp: "Formatted at 2026-09-23 21:00:24 on dist-test-slave-8xn5"
server_key: "394d8061392710334340349a081540b0"
server_key_iv: "03b3fb1c7083309bed883cfcf7905f48"
server_key_version: "encryptionkey@0"
I20260923 21:00:24.913975 2356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:24.933949 2356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:24.938405 2356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:24.938730 2356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:24.941758 2356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:24.941890 2356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:24.942045 2356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:24.942173 2356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:00:24 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:24.969097 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.942308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42911 (local address 127.0.97.62:37889)
0923 21:00:24.942567 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:24.942576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:24.944244 (+ 1668us) server_negotiation.cc:407] Connection header received
0923 21:00:24.946722 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:24.946728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:24.946820 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:24.946948 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:24.949475 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.950598 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.953259 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.953843 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.953987 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.959255 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:24.959264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:24.959274 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:24.959311 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:24.963263 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.964377 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.964386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.964392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.964601 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.965335 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.965343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.965349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.965857 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:24.966048 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:24.968565 (+ 2517us) server_negotiation.cc:299] Negotiation successful
0923 21:00:24.968815 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 21:00:24.970893 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.943229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37889 (local address 127.0.97.2:42911)
0923 21:00:24.944073 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:24.944117 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:24.946406 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:24.947296 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:24.947312 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:24.948195 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:24.949167 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.949202 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.950788 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.950792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.953100 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.953114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.954112 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.954119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.954408 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.955258 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:24.955283 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:24.958922 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:24.963504 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.963521 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.963548 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.964228 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.964898 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.964902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.964904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.965201 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.966105 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:24.966116 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:24.966702 (+ 586us) client_negotiation.cc:769] Sending connection context
0923 21:00:24.967195 (+ 493us) client_negotiation.cc:240] Negotiation successful
0923 21:00:24.967972 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":375,"threads_started":1}
I20260923 21:00:24.993598 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.976752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52293 (local address 127.0.97.62:37889)
0923 21:00:24.976928 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:24.976932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:24.977924 (+ 992us) server_negotiation.cc:407] Connection header received
0923 21:00:24.978187 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:24.978196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:24.978307 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:24.978606 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:24.979539 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.981027 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.982646 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.983505 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.983687 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.986747 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:24.986754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:24.986761 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:24.986805 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:24.990262 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.991308 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.991317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.991321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.991512 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:24.992048 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:24.992051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:24.992054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:24.992384 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:24.992533 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:24.993148 (+ 615us) server_negotiation.cc:299] Negotiation successful
0923 21:00:24.993325 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 21:00:24.994346 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:24.976846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37889 (local address 127.0.97.2:52293)
0923 21:00:24.977733 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:24.977797 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:24.978004 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:24.978584 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:24.978587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:24.979090 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:24.979377 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.979386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.981248 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.981253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.982459 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:24.982480 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:24.983979 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:24.983993 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:24.984139 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:24.984896 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:24.984933 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:24.986368 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:24.990489 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.990495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.990500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.991166 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.991666 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:24.991675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:24.991682 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:24.991902 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:24.992547 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:24.992555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:24.992862 (+ 307us) client_negotiation.cc:769] Sending connection context
0923 21:00:24.993062 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 21:00:24.993599 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":630,"threads_started":1}
I20260923 21:00:25.000015 2356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37847
I20260923 21:00:25.000057 2490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37847 every 8 connection(s)
I20260923 21:00:25.001353 2356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:25.003715 2491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37889
I20260923 21:00:25.003927 2491 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:25.004586 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 2356
I20260923 21:00:25.004765 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:25.004772 2491 heartbeater.cc:511] Master 127.0.97.62:37889 requested a full tablet report, sending...
I20260923 21:00:25.005338 388 external_mini_cluster.cc:1428] Setting key 1367aa4b130d3a19696a1eb0223f6a9a
I20260923 21:00:25.006685 2144 ts_manager.cc:194] Registered new tserver with Master: d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:25.007728 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42911
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:25.051118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37889
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:25.391400 2495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:25.391693 2495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:25.391804 2495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:25.399325 2495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:25.399444 2495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:25.399533 2495 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 21:00:25.399585 2495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:00:25.399647 2495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:25.409869 2495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:25.411275 2495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:25.412632 2495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:25.427083 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:25.427158 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:25.427120 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:25.428241 2495 server_base.cc:1034] running on GCE node
I20260923 21:00:25.428992 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:25.430094 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:25.431309 2495 hybrid_clock.cc:648] HybridClock initialized: now 1790197225431239 us; error 74 us; skew 500 ppm
I20260923 21:00:25.431674 2495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:25 dist-test-slave-8xn5 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:25.435256 2495 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:25.436906 2495 webserver.cc:460] Webserver started at http://127.0.97.3:45911/ using document root <none> and password file <none>
I20260923 21:00:25.437675 2495 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:25.437836 2495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:25.438171 2495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:25.441437 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "32808b1db26c45a195b4e1f56517e20b"
format_stamp: "Formatted at 2026-09-23 21:00:25 on dist-test-slave-8xn5"
server_key: "b0df89d1a3b756dc395f17b573369db8"
server_key_iv: "79a73f53ed5d1865570aa32be2858147"
server_key_version: "encryptionkey@0"
I20260923 21:00:25.442430 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "32808b1db26c45a195b4e1f56517e20b"
format_stamp: "Formatted at 2026-09-23 21:00:25 on dist-test-slave-8xn5"
server_key: "b0df89d1a3b756dc395f17b573369db8"
server_key_iv: "79a73f53ed5d1865570aa32be2858147"
server_key_version: "encryptionkey@0"
I20260923 21:00:25.448135 2495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 21:00:25.451964 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:25.453474 2495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:00:25.453672 2495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "32808b1db26c45a195b4e1f56517e20b"
format_stamp: "Formatted at 2026-09-23 21:00:25 on dist-test-slave-8xn5"
server_key: "b0df89d1a3b756dc395f17b573369db8"
server_key_iv: "79a73f53ed5d1865570aa32be2858147"
server_key_version: "encryptionkey@0"
I20260923 21:00:25.453905 2495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:25.470979 2495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:25.473799 2352 heartbeater.cc:503] Master 127.0.97.62:37889 was elected leader, sending a full tablet report...
I20260923 21:00:25.476341 2495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:25.476641 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:25.479525 2495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:25.479677 2495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:25.479809 2495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:25.479923 2495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:00:25 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:25.509642 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.480122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52387 (local address 127.0.97.62:37889)
0923 21:00:25.480317 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:25.480323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:25.481928 (+ 1605us) server_negotiation.cc:407] Connection header received
0923 21:00:25.484438 (+ 2510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:25.484444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:25.484549 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:25.484704 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:25.487596 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.488955 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.492602 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.493447 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.493615 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.500035 (+ 6420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:25.500045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:25.500053 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:25.500113 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:25.504873 (+ 4760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:25.505842 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:25.505850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:25.505857 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:25.506064 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:25.506963 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:25.506972 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:25.506982 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:25.507486 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:25.507702 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:25.508928 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0923 21:00:25.509199 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 21:00:25.511473 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.480812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37889 (local address 127.0.97.3:52387)
0923 21:00:25.481713 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:25.481764 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:25.484044 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:25.485120 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:25.485137 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:25.486120 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:25.487267 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.487298 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.489234 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.489247 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.492296 (+ 3049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.492321 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.494025 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.494037 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.494534 (+ 497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.495719 (+ 1185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:25.495762 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:25.499677 (+ 3915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:25.505117 (+ 5440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:25.505126 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:25.505144 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:25.505673 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:25.506245 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:25.506251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:25.506256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:25.506652 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:25.507670 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:25.507681 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:25.508303 (+ 622us) client_negotiation.cc:769] Sending connection context
0923 21:00:25.508798 (+ 495us) client_negotiation.cc:240] Negotiation successful
0923 21:00:25.509472 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":387,"threads_started":1}
I20260923 21:00:25.534142 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.519635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37889 (local address 127.0.97.3:43539)
0923 21:00:25.520410 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:25.520433 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:25.520596 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:25.521015 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:25.521019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:25.521529 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:25.521814 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.521824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.522963 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.522969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.524017 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.524029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.524787 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.524793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.524982 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.525663 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:25.525681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:25.526567 (+ 886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:25.530781 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:25.530787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:25.530791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:25.531602 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:25.532147 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:25.532152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:25.532155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:25.532321 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:25.532998 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:25.533003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:25.533253 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 21:00:25.533451 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 21:00:25.533793 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":462,"threads_started":1}
I20260923 21:00:25.534605 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.518931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43539 (local address 127.0.97.62:37889)
0923 21:00:25.519144 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:25.519153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:25.520532 (+ 1379us) server_negotiation.cc:407] Connection header received
0923 21:00:25.520738 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:25.520744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:25.520831 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:25.520994 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:25.521977 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.522793 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.524176 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.524590 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.524737 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.526772 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:25.526777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:25.526780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:25.526812 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:25.530536 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:25.531770 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:25.531778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:25.531784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:25.531980 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:25.532444 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:25.532458 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:25.532463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:25.532869 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:25.533021 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:25.533896 (+ 875us) server_negotiation.cc:299] Negotiation successful
0923 21:00:25.534122 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 21:00:25.536273 2495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35165
I20260923 21:00:25.536293 2629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35165 every 8 connection(s)
I20260923 21:00:25.537741 2495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:25.540464 2630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37889
I20260923 21:00:25.540732 2630 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:25.541679 2630 heartbeater.cc:511] Master 127.0.97.62:37889 requested a full tablet report, sending...
I20260923 21:00:25.542050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 2495
I20260923 21:00:25.542222 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:25.542791 388 external_mini_cluster.cc:1428] Setting key 9af5a3fb899d7cf613753d9f591cb792
I20260923 21:00:25.543411 2144 ts_manager.cc:194] Registered new tserver with Master: 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165)
I20260923 21:00:25.544421 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52387
I20260923 21:00:25.545982 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 21:00:25 dist-test-slave-8xn5 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 21:00:25 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:25.595978 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.580727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.97.62:37889)
0923 21:00:25.581006 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:25.581010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:25.581355 (+ 345us) server_negotiation.cc:407] Connection header received
0923 21:00:25.581502 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:25.581506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:25.581578 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:25.581687 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:25.583330 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.584178 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.585605 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.586150 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.586282 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.590458 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:25.590464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:25.590467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:25.590535 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:25.593148 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:25.593868 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:25.593873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:25.593875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:25.594004 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:25.594621 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:25.594624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:25.594627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:25.594971 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:25.595172 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:25.595452 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 21:00:25.595622 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 21:00:25.600582 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:25.607599 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:25.635890 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.620899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37979 (local address 127.0.0.1:46056)
0923 21:00:25.622575 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:25.622614 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:25.622787 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:25.624432 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:25.624444 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:25.624485 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:25.625009 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.625026 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.627952 (+ 2926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.627961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.631418 (+ 3457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.631442 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.634522 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.634534 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.634781 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.634844 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 21:00:25.634996 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 21:00:25.635215 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"mutex_wait_us":5,"thread_start_us":1164,"threads_started":1}
I20260923 21:00:25.636328 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.621035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35165 (local address 127.0.0.1:40930)
0923 21:00:25.623415 (+ 2380us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:25.623435 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:25.623546 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:25.626005 (+ 2459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:25.626013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:25.626050 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:25.626576 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.626593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.629367 (+ 2774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.629385 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.632603 (+ 3218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.632626 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.635505 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.635513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.635719 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.635754 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:00:25.635827 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 21:00:25.635979 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2237,"thread_start_us":2062,"threads_started":1}
I20260923 21:00:25.637423 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.621054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.0.97.3:35165)
0923 21:00:25.624859 (+ 3805us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:25.624872 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:25.624911 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:00:25.625038 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:25.625049 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:25.625388 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:25.625622 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:25.626853 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.629117 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.632926 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.635172 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.636541 (+ 1369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.636814 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:25.636906 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 21:00:25.637095 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3586,"thread_start_us":3377,"threads_started":1}
I20260923 21:00:25.637792 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.621596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.0.97.1:37979)
0923 21:00:25.623695 (+ 2099us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:25.623708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:25.623747 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:00:25.623855 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:25.623863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:25.624153 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:25.625304 (+ 1151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:25.625576 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.627719 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.631749 (+ 4030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.634079 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.636995 (+ 2916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.637243 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:25.637313 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 21:00:25.637496 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1923,"thread_start_us":1798,"threads_started":1}
I20260923 21:00:25.641779 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5a13efb5174ce1a9619724449fa448 (DEFAULT_TABLE table=test-table [id=101efd795160462686a9853766fa5f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:25.641788 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5a13efb5174ce1a9619724449fa448 (DEFAULT_TABLE table=test-table [id=101efd795160462686a9853766fa5f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:25.643232 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5a13efb5174ce1a9619724449fa448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:25.643332 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5a13efb5174ce1a9619724449fa448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:25.645237 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.623599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37847 (local address 127.0.0.1:55818)
0923 21:00:25.635704 (+ 12105us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:25.635725 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:25.635885 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:25.636731 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:25.636735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:25.636764 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:25.637094 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.637105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.639864 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.639872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.642004 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:25.642027 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.644340 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.644347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:25.644589 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.644625 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 21:00:25.644737 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 21:00:25.644896 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11959,"thread_start_us":11764,"threads_started":1}
I20260923 21:00:25.646008 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:25.623964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.0.97.2:37847)
0923 21:00:25.634989 (+ 11025us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:25.634999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:25.636035 (+ 1036us) server_negotiation.cc:407] Connection header received
0923 21:00:25.636161 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:25.636171 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:25.636502 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:25.636702 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:25.637289 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.639661 (+ 2372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.642210 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:25.643809 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:25.643963 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:25.644168 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:25.645545 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0923 21:00:25.645722 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10796,"thread_start_us":11040,"threads_started":1}
I20260923 21:00:25.650271 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 9a5a13efb5174ce1a9619724449fa448 (DEFAULT_TABLE table=test-table [id=101efd795160462686a9853766fa5f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:25.651872 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a5a13efb5174ce1a9619724449fa448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:25.657728 2649 tablet_bootstrap.cc:492] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Bootstrap starting.
I20260923 21:00:25.660382 2650 tablet_bootstrap.cc:492] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Bootstrap starting.
I20260923 21:00:25.662010 2649 tablet_bootstrap.cc:654] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:25.663539 2649 log.cc:824] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:25.664346 2650 tablet_bootstrap.cc:654] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:25.666113 2650 log.cc:824] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:25.666909 2652 tablet_bootstrap.cc:492] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Bootstrap starting.
I20260923 21:00:25.670590 2652 tablet_bootstrap.cc:654] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:25.670691 2649 tablet_bootstrap.cc:492] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: No bootstrap required, opened a new log
I20260923 21:00:25.670770 2650 tablet_bootstrap.cc:492] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: No bootstrap required, opened a new log
I20260923 21:00:25.671018 2649 ts_tablet_manager.cc:1397] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260923 21:00:25.671058 2650 ts_tablet_manager.cc:1397] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260923 21:00:25.671914 2652 log.cc:824] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:25.676505 2652 tablet_bootstrap.cc:492] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: No bootstrap required, opened a new log
I20260923 21:00:25.676827 2652 ts_tablet_manager.cc:1397] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260923 21:00:25.679613 2649 raft_consensus.cc:348] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:25.680045 2649 raft_consensus.cc:374] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:25.680145 2649 raft_consensus.cc:729] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Initialized, Role: FOLLOWER
I20260923 21:00:25.680812 2649 consensus_queue.cc:260] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:25.682068 2630 heartbeater.cc:503] Master 127.0.97.62:37889 was elected leader, sending a full tablet report...
I20260923 21:00:25.681944 2650 raft_consensus.cc:348] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:25.682509 2650 raft_consensus.cc:374] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:25.682600 2650 raft_consensus.cc:729] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ee0b4e519b4dc9b2b8aa0850646a1d, State: Initialized, Role: FOLLOWER
I20260923 21:00:25.683391 2649 ts_tablet_manager.cc:1428] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260923 21:00:25.683290 2650 consensus_queue.cc:260] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:25.685267 2650 ts_tablet_manager.cc:1428] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260923 21:00:25.690475 2652 raft_consensus.cc:348] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:25.690864 2652 raft_consensus.cc:374] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:25.690991 2652 raft_consensus.cc:729] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247ea347bbb47a3a6ba3bfe1c42e9ea, State: Initialized, Role: FOLLOWER
I20260923 21:00:25.691561 2652 consensus_queue.cc:260] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:25.692534 2491 heartbeater.cc:503] Master 127.0.97.62:37889 was elected leader, sending a full tablet report...
I20260923 21:00:25.693202 2652 ts_tablet_manager.cc:1428] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260923 21:00:25.714191 2353 tablet.cc:2369] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:00:25.753438 2492 tablet.cc:2369] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:00:25.790061 2631 tablet.cc:2369] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:00:26.005690 2656 raft_consensus.cc:482] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:26.006042 2656 raft_consensus.cc:504] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:26.007961 2656 leader_election.cc:288] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:26.017930 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.009392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58493 (local address 127.0.97.2:37847)
0923 21:00:26.009594 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.009600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.010471 (+ 871us) server_negotiation.cc:407] Connection header received
0923 21:00:26.010707 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.010712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.010789 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.010883 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:26.011662 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.012801 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.015799 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.016595 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.016789 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.016945 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.017370 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.017632 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 21:00:26.017994 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.008237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37847 (local address 127.0.97.1:58493)
0923 21:00:26.010359 (+ 2122us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.010377 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.010517 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.011039 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.011043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.011072 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:26.011431 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.011440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.012946 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.012951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.015490 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.015521 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.016900 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.016908 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.017122 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.017172 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.017293 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.017481 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1970,"mutex_wait_us":49,"thread_start_us":1797,"threads_started":1}
I20260923 21:00:26.018059 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.008201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35165 (local address 127.0.97.1:34381)
0923 21:00:26.010141 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.010164 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.010483 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.010873 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.010878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.010909 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:26.011389 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.011405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.012458 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.012464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.015490 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.015521 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.017239 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.017247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.017437 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.017468 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.017555 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.017694 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"mutex_wait_us":32,"spinlock_wait_cycles":33536,"thread_start_us":1472,"threads_started":1}
I20260923 21:00:26.018280 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.008731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34381 (local address 127.0.97.3:35165)
0923 21:00:26.009022 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.009032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.010284 (+ 1252us) server_negotiation.cc:407] Connection header received
0923 21:00:26.010650 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.010656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.010733 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.010834 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:26.011562 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.012312 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.015759 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.016987 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.017142 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.017236 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.017634 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.017928 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 21:00:26.019588 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a13efb5174ce1a9619724449fa448" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32808b1db26c45a195b4e1f56517e20b" is_pre_election: true
I20260923 21:00:26.019588 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a13efb5174ce1a9619724449fa448" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" is_pre_election: true
I20260923 21:00:26.020139 2580 raft_consensus.cc:2454] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 0.
I20260923 21:00:26.020126 2441 raft_consensus.cc:2454] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 0.
I20260923 21:00:26.020915 2235 leader_election.cc:302] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, d247ea347bbb47a3a6ba3bfe1c42e9ea, e0ee0b4e519b4dc9b2b8aa0850646a1d; no voters:
I20260923 21:00:26.021471 2656 raft_consensus.cc:2781] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:26.021628 2656 raft_consensus.cc:482] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:26.021736 2656 raft_consensus.cc:3037] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.024202 2656 raft_consensus.cc:504] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:26.024864 2656 leader_election.cc:288] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 election: Requested vote from peers 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:26.025457 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a13efb5174ce1a9619724449fa448" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32808b1db26c45a195b4e1f56517e20b"
I20260923 21:00:26.025511 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a5a13efb5174ce1a9619724449fa448" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
I20260923 21:00:26.025699 2580 raft_consensus.cc:3037] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.025731 2441 raft_consensus.cc:3037] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.028093 2441 raft_consensus.cc:2454] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 1.
I20260923 21:00:26.028126 2580 raft_consensus.cc:2454] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 1.
I20260923 21:00:26.028674 2232 leader_election.cc:302] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d247ea347bbb47a3a6ba3bfe1c42e9ea, e0ee0b4e519b4dc9b2b8aa0850646a1d; no voters:
I20260923 21:00:26.029081 2656 raft_consensus.cc:2781] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:26.029613 2656 raft_consensus.cc:686] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 LEADER]: Becoming Leader. State: Replica: e0ee0b4e519b4dc9b2b8aa0850646a1d, State: Running, Role: LEADER
I20260923 21:00:26.030198 2656 consensus_queue.cc:237] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } }
I20260923 21:00:26.035077 2145 catalog_manager.cc:5495] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } health_report { overall_health: UNKNOWN } } }
Sep 23 21:00:26 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:26.131170 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.116860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.97.62:37889)
0923 21:00:26.117456 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.117460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.117919 (+ 459us) server_negotiation.cc:407] Connection header received
0923 21:00:26.118049 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.118052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.118110 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.118196 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:26.119457 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.120330 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.121584 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.121981 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.122076 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.125531 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:26.125536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:26.125539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:26.125568 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:26.128267 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:26.129020 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:26.129028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:26.129031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:26.129161 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:26.129532 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:26.129540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:26.129543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:26.129806 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:26.129963 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.130437 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.130810 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":440,"thread_start_us":285,"threads_started":1}
I20260923 21:00:26.131235 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.116880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37889 (local address 127.0.0.1:42658)
0923 21:00:26.117728 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.117757 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.117890 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.118242 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.118245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.119035 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:26.119296 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.119304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.120467 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.120471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.121449 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.121459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.122151 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.122155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.122293 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.123021 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:26.123038 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:26.125294 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:26.128428 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:26.128442 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:26.128451 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:26.128896 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:26.129279 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:26.129283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:26.129286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:26.129429 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:26.129939 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:26.129947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:26.130142 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.130323 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.130681 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":1,"thread_start_us":567,"threads_started":1}
I20260923 21:00:26.137042 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42658:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 21:00:26.137992 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:26.144981 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 673aa9dec5ff4093a9b2d358cb1a13a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e636a1cfc3747f295a6b357b412e22f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:00:26.145335 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 673aa9dec5ff4093a9b2d358cb1a13a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e636a1cfc3747f295a6b357b412e22f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:00:26.146040 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673aa9dec5ff4093a9b2d358cb1a13a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:26.145650 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 673aa9dec5ff4093a9b2d358cb1a13a6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e636a1cfc3747f295a6b357b412e22f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:00:26.146075 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673aa9dec5ff4093a9b2d358cb1a13a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:26.146382 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 673aa9dec5ff4093a9b2d358cb1a13a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:26.152297 2649 tablet_bootstrap.cc:492] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: Bootstrap starting.
I20260923 21:00:26.153199 2650 tablet_bootstrap.cc:492] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Bootstrap starting.
I20260923 21:00:26.153199 2652 tablet_bootstrap.cc:492] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Bootstrap starting.
I20260923 21:00:26.154110 2649 tablet_bootstrap.cc:654] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:26.154858 2652 tablet_bootstrap.cc:654] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:26.154858 2650 tablet_bootstrap.cc:654] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:26.156503 2649 tablet_bootstrap.cc:492] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: No bootstrap required, opened a new log
I20260923 21:00:26.156716 2649 ts_tablet_manager.cc:1397] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260923 21:00:26.157289 2650 tablet_bootstrap.cc:492] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: No bootstrap required, opened a new log
I20260923 21:00:26.157464 2650 ts_tablet_manager.cc:1397] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:26.157641 2649 raft_consensus.cc:348] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.157918 2649 raft_consensus.cc:374] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:26.158027 2649 raft_consensus.cc:729] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Initialized, Role: FOLLOWER
I20260923 21:00:26.158275 2649 consensus_queue.cc:260] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.158253 2650 raft_consensus.cc:348] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.158533 2650 raft_consensus.cc:374] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:26.159024 2649 ts_tablet_manager.cc:1428] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 21:00:26.159067 2655 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.159410 2655 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:26.159474 2652 tablet_bootstrap.cc:492] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: No bootstrap required, opened a new log
I20260923 21:00:26.159607 2652 ts_tablet_manager.cc:1397] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 21:00:26.159715 2650 raft_consensus.cc:729] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ee0b4e519b4dc9b2b8aa0850646a1d, State: Initialized, Role: FOLLOWER
I20260923 21:00:26.160013 2650 consensus_queue.cc:260] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.160352 2652 raft_consensus.cc:348] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.160601 2650 ts_tablet_manager.cc:1428] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 21:00:26.160439 2656 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.160666 2652 raft_consensus.cc:374] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:26.160769 2652 raft_consensus.cc:729] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247ea347bbb47a3a6ba3bfe1c42e9ea, State: Initialized, Role: FOLLOWER
I20260923 21:00:26.160789 2656 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:26.161031 2652 consensus_queue.cc:260] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.161607 2652 ts_tablet_manager.cc:1428] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 21:00:26.161780 2657 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.162187 2657 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:26.329200 2655 raft_consensus.cc:482] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:26.329473 2655 raft_consensus.cc:504] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.331110 2655 leader_election.cc:288] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:26.337581 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.331699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38213 (local address 127.0.97.1:37979)
0923 21:00:26.332230 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.332237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.332480 (+ 243us) server_negotiation.cc:407] Connection header received
0923 21:00:26.332702 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.332710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.332814 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.332919 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:26.333425 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.334173 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.335771 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.336570 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.336720 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.336821 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.337086 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.337226 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":269,"threads_started":1}
I20260923 21:00:26.337590 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.331389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37979 (local address 127.0.97.3:38213)
0923 21:00:26.332368 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.332385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.332554 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.332975 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.332979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.333003 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:26.333250 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.333258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.334305 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.334313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.335606 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.335617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.336739 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.336745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.336923 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.336947 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.337055 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.337205 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":1383,"threads_started":1}
I20260923 21:00:26.338317 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.331389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37847 (local address 127.0.97.3:58729)
0923 21:00:26.333087 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.333108 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.333265 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.333603 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.333607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.333628 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:26.333930 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.333938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.334992 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.334997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.336311 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.336322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.337433 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.337439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.337606 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.337629 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.337850 (+ 221us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.337969 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577,"mutex_wait_us":27,"thread_start_us":711,"threads_started":1}
I20260923 21:00:26.338510 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.331552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58729 (local address 127.0.97.2:37847)
0923 21:00:26.331746 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.331752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.333204 (+ 1452us) server_negotiation.cc:407] Connection header received
0923 21:00:26.333417 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.333421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.333481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.333587 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:26.334072 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.334856 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.336460 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.337212 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.337368 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.337468 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.338044 (+ 576us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.338193 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:00:26.338601 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673aa9dec5ff4093a9b2d358cb1a13a6" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" is_pre_election: true
I20260923 21:00:26.339026 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673aa9dec5ff4093a9b2d358cb1a13a6" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" is_pre_election: true
I20260923 21:00:26.339061 2302 raft_consensus.cc:2454] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 0.
I20260923 21:00:26.339257 2441 raft_consensus.cc:2454] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 0.
I20260923 21:00:26.339741 2511 leader_election.cc:302] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, d247ea347bbb47a3a6ba3bfe1c42e9ea, e0ee0b4e519b4dc9b2b8aa0850646a1d; no voters:
I20260923 21:00:26.340165 2655 raft_consensus.cc:2781] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:26.340317 2655 raft_consensus.cc:482] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:26.340425 2655 raft_consensus.cc:3037] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.343041 2655 raft_consensus.cc:504] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.343675 2655 leader_election.cc:288] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 election: Requested vote from peers e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:26.344202 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673aa9dec5ff4093a9b2d358cb1a13a6" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d"
I20260923 21:00:26.344269 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "673aa9dec5ff4093a9b2d358cb1a13a6" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
I20260923 21:00:26.344451 2302 raft_consensus.cc:3037] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.344480 2441 raft_consensus.cc:3037] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.347554 2441 raft_consensus.cc:2454] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 1.
I20260923 21:00:26.347568 2302 raft_consensus.cc:2454] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 1.
I20260923 21:00:26.348091 2512 leader_election.cc:302] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, e0ee0b4e519b4dc9b2b8aa0850646a1d; no voters:
I20260923 21:00:26.348474 2655 raft_consensus.cc:2781] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:26.349054 2655 raft_consensus.cc:686] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [term 1 LEADER]: Becoming Leader. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Running, Role: LEADER
I20260923 21:00:26.349752 2655 consensus_queue.cc:237] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.353439 2675 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: TxnStatusTablet state changed. Reason: New leader 32808b1db26c45a195b4e1f56517e20b. Latest consensus state: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.353912 2675 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:00:26.354748 2677 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 21:00:26.355072 2677 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 21:00:26.355072 2145 catalog_manager.cc:5495] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b reported cstate change: term changed from 0 to 1, leader changed from <none> to 32808b1db26c45a195b4e1f56517e20b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } health_report { overall_health: HEALTHY } } }
I20260923 21:00:26.403154 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.395832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35165 (local address 127.0.0.1:40944)
0923 21:00:26.396114 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.396136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.396313 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.396821 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.396827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.396841 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:00:26.397049 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.397060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.398376 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.398383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.400063 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.400079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.401288 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.401295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.401484 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.401569 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.402567 (+ 998us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.402589 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.402667 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.402840 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260923 21:00:26.403183 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.396119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.0.97.3:35165)
0923 21:00:26.396314 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.396320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.396364 (+ 44us) server_negotiation.cc:407] Connection header received
0923 21:00:26.396534 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.396538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.396633 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.396771 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:26.397244 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.398209 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.400260 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.401107 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.401257 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.401741 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.402431 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.402701 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.402753 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.402836 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 21:00:26.411376 2441 raft_consensus.cc:1261] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Refusing update from remote peer 32808b1db26c45a195b4e1f56517e20b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:26.411397 2302 raft_consensus.cc:1261] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Refusing update from remote peer 32808b1db26c45a195b4e1f56517e20b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:26.412374 2675 consensus_queue.cc:1035] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:26.412731 2655 consensus_queue.cc:1035] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:26.423768 2656 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: TxnStatusTablet state changed. Reason: New leader 32808b1db26c45a195b4e1f56517e20b. Latest consensus state: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.424146 2656 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:26.425900 2657 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: TxnStatusTablet state changed. Reason: New leader 32808b1db26c45a195b4e1f56517e20b. Latest consensus state: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.426209 2657 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:26.427553 2680 mvcc.cc:205] Tried to move back new op lower bound from 7332647839371001856 to 7332647839133528064. Current Snapshot: MvccSnapshot[applied={T|T < 7332647839371001856}]
I20260923 21:00:26.439375 2656 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.439683 2656 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P e0ee0b4e519b4dc9b2b8aa0850646a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:26.440150 2657 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.440416 2657 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P d247ea347bbb47a3a6ba3bfe1c42e9ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:00:26.442391 2675 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.442732 2675 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:00:26.443179 2675 tablet_replica.cc:439] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } } }
I20260923 21:00:26.443440 2675 tablet_replica.cc:442] T 673aa9dec5ff4093a9b2d358cb1a13a6 P 32808b1db26c45a195b4e1f56517e20b: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:00:26.456257 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.450176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.97.1:37979)
0923 21:00:26.450404 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.450410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.450950 (+ 540us) server_negotiation.cc:407] Connection header received
0923 21:00:26.451209 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.451213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.451305 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.451510 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:26.451955 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.452668 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.454078 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.454542 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.454675 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.455088 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.455661 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.455818 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.455891 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.455973 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 21:00:26.478796 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.470207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59385 (local address 127.0.97.3:35165)
0923 21:00:26.470475 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.470483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.471359 (+ 876us) server_negotiation.cc:407] Connection header received
0923 21:00:26.471577 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.471583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.471688 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.471831 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:26.472474 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.473713 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.475941 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.476960 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.477129 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.477765 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.477939 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.478056 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.478259 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.478394 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 21:00:26.478873 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.470559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35165 (local address 127.0.97.1:59385)
0923 21:00:26.471222 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.471242 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.471411 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.471847 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.471851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.471864 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:00:26.472229 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.472240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.473872 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.473878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.475749 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.475764 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.477201 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.477210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.477405 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.477489 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.478089 (+ 600us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.478103 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.478214 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.478458 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":334,"threads_started":1}
I20260923 21:00:26.490024 2441 raft_consensus.cc:1261] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Refusing update from remote peer e0ee0b4e519b4dc9b2b8aa0850646a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:26.490656 2580 raft_consensus.cc:1261] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Refusing update from remote peer e0ee0b4e519b4dc9b2b8aa0850646a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:26.490828 2660 consensus_queue.cc:1035] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:26.491441 2656 consensus_queue.cc:1035] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:26.496362 2681 mvcc.cc:205] Tried to move back new op lower bound from 7332647839696470016 to 7332647837823971328. Current Snapshot: MvccSnapshot[applied={T|T < 7332647839696470016}]
I20260923 21:00:26.535701 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.528550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56493 (local address 127.0.97.1:37979)
0923 21:00:26.528725 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.528731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.529963 (+ 1232us) server_negotiation.cc:407] Connection header received
0923 21:00:26.530200 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.530207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.530299 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.530405 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:26.531160 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.531901 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.533549 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.534142 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.534284 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.534916 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.535041 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.535139 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.535299 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.535387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 21:00:26.536098 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.528609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37979 (local address 127.0.97.3:56493)
0923 21:00:26.529826 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.529845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.529992 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.530649 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.530652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.530662 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:00:26.530992 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.531001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.532041 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.532046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.533409 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.533421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.534484 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.534488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.534611 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.534723 (+ 112us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.535138 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.535150 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.535257 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.535408 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":827,"threads_started":1}
I20260923 21:00:26.583539 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:26.583808 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:26.587790 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Sending DeleteTablet for 3 replicas of tablet 9a5a13efb5174ce1a9619724449fa448
I20260923 21:00:26.589425 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5a13efb5174ce1a9619724449fa448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55818
I20260923 21:00:26.590200 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5a13efb5174ce1a9619724449fa448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40930
I20260923 21:00:26.590601 2700 tablet_replica.cc:330] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: stopping tablet replica
I20260923 21:00:26.591071 2700 raft_consensus.cc:2229] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:26.591558 2700 raft_consensus.cc:2258] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:26.592124 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a5a13efb5174ce1a9619724449fa448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46056
I20260923 21:00:26.593175 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:26.593915 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:26.597198 2702 tablet_replica.cc:330] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: stopping tablet replica
I20260923 21:00:26.597751 2702 raft_consensus.cc:2229] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:26.598465 2702 raft_consensus.cc:2258] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:26.602931 2282 tablet_service.cc:1467] Processing CreateTablet for tablet f4a622d6a6d94e9281e8a15bbd51a2d2 (DEFAULT_TABLE table=test-table [id=ebed19d64c0c43d59a13519f5ebea1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:26.603379 2559 tablet_service.cc:1467] Processing CreateTablet for tablet f4a622d6a6d94e9281e8a15bbd51a2d2 (DEFAULT_TABLE table=test-table [id=ebed19d64c0c43d59a13519f5ebea1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:26.604020 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a622d6a6d94e9281e8a15bbd51a2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:26.604723 2421 tablet_service.cc:1467] Processing CreateTablet for tablet f4a622d6a6d94e9281e8a15bbd51a2d2 (DEFAULT_TABLE table=test-table [id=ebed19d64c0c43d59a13519f5ebea1da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:26.605062 2700 ts_tablet_manager.cc:1905] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:26.605208 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a622d6a6d94e9281e8a15bbd51a2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:26.603436 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a622d6a6d94e9281e8a15bbd51a2d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:26.611382 2649 tablet_bootstrap.cc:492] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: Bootstrap starting.
I20260923 21:00:26.613232 2649 tablet_bootstrap.cc:654] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:26.613480 2700 ts_tablet_manager.cc:1918] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:26.613727 2700 log.cc:1196] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/9a5a13efb5174ce1a9619724449fa448
I20260923 21:00:26.614615 2700 ts_tablet_manager.cc:1939] T 9a5a13efb5174ce1a9619724449fa448 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting consensus metadata
I20260923 21:00:26.617002 2129 catalog_manager.cc:4841] TS d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847): tablet 9a5a13efb5174ce1a9619724449fa448 (table test-table [id=101efd795160462686a9853766fa5f34]) successfully deleted
I20260923 21:00:26.619663 2702 ts_tablet_manager.cc:1905] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:26.622493 2652 tablet_bootstrap.cc:492] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Bootstrap starting.
I20260923 21:00:26.624321 2652 tablet_bootstrap.cc:654] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:26.627130 2649 tablet_bootstrap.cc:492] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: No bootstrap required, opened a new log
I20260923 21:00:26.627393 2649 ts_tablet_manager.cc:1397] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20260923 21:00:26.628185 2702 ts_tablet_manager.cc:1918] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:26.628381 2702 log.cc:1196] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/9a5a13efb5174ce1a9619724449fa448
I20260923 21:00:26.628415 2649 raft_consensus.cc:348] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.628844 2649 raft_consensus.cc:374] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:26.628957 2702 ts_tablet_manager.cc:1939] T 9a5a13efb5174ce1a9619724449fa448 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting consensus metadata
I20260923 21:00:26.628991 2649 raft_consensus.cc:729] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Initialized, Role: FOLLOWER
I20260923 21:00:26.629388 2649 consensus_queue.cc:260] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.630234 2649 ts_tablet_manager.cc:1428] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260923 21:00:26.630913 2130 catalog_manager.cc:4841] TS e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979): tablet 9a5a13efb5174ce1a9619724449fa448 (table test-table [id=101efd795160462686a9853766fa5f34]) successfully deleted
I20260923 21:00:26.631896 2650 tablet_bootstrap.cc:492] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Bootstrap starting.
I20260923 21:00:26.633688 2650 tablet_bootstrap.cc:654] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:26.640237 2701 tablet_replica.cc:330] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: stopping tablet replica
I20260923 21:00:26.640475 2650 tablet_bootstrap.cc:492] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: No bootstrap required, opened a new log
I20260923 21:00:26.640641 2650 ts_tablet_manager.cc:1397] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260923 21:00:26.640710 2701 raft_consensus.cc:2229] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:26.641040 2701 raft_consensus.cc:2258] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:26.641075 2652 tablet_bootstrap.cc:492] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: No bootstrap required, opened a new log
I20260923 21:00:26.641412 2652 ts_tablet_manager.cc:1397] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20260923 21:00:26.641455 2650 raft_consensus.cc:348] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.641713 2650 raft_consensus.cc:374] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:26.641812 2650 raft_consensus.cc:729] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ee0b4e519b4dc9b2b8aa0850646a1d, State: Initialized, Role: FOLLOWER
I20260923 21:00:26.642082 2650 consensus_queue.cc:260] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.642767 2650 ts_tablet_manager.cc:1428] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 21:00:26.643392 2701 ts_tablet_manager.cc:1905] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:26.645634 2652 raft_consensus.cc:348] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.647415 2652 raft_consensus.cc:374] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:26.647523 2652 raft_consensus.cc:729] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247ea347bbb47a3a6ba3bfe1c42e9ea, State: Initialized, Role: FOLLOWER
I20260923 21:00:26.647845 2652 consensus_queue.cc:260] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.650915 2652 ts_tablet_manager.cc:1428] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20260923 21:00:26.653299 2701 ts_tablet_manager.cc:1918] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:26.653542 2701 log.cc:1196] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/9a5a13efb5174ce1a9619724449fa448
I20260923 21:00:26.654206 2701 ts_tablet_manager.cc:1939] T 9a5a13efb5174ce1a9619724449fa448 P 32808b1db26c45a195b4e1f56517e20b: Deleting consensus metadata
I20260923 21:00:26.656373 2132 catalog_manager.cc:4841] TS 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165): tablet 9a5a13efb5174ce1a9619724449fa448 (table test-table [id=101efd795160462686a9853766fa5f34]) successfully deleted
I20260923 21:00:26.773551 2675 raft_consensus.cc:482] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:26.773818 2675 raft_consensus.cc:504] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.774682 2675 leader_election.cc:288] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:26.775439 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a622d6a6d94e9281e8a15bbd51a2d2" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" is_pre_election: true
I20260923 21:00:26.775403 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a622d6a6d94e9281e8a15bbd51a2d2" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" is_pre_election: true
I20260923 21:00:26.775683 2441 raft_consensus.cc:2454] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 0.
I20260923 21:00:26.775717 2302 raft_consensus.cc:2454] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 0.
I20260923 21:00:26.776175 2511 leader_election.cc:302] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, d247ea347bbb47a3a6ba3bfe1c42e9ea; no voters:
I20260923 21:00:26.776522 2675 raft_consensus.cc:2781] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:26.776646 2675 raft_consensus.cc:482] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:26.776764 2675 raft_consensus.cc:3037] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.780134 2675 raft_consensus.cc:504] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.781490 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a622d6a6d94e9281e8a15bbd51a2d2" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
I20260923 21:00:26.781498 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a622d6a6d94e9281e8a15bbd51a2d2" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d"
I20260923 21:00:26.781724 2441 raft_consensus.cc:3037] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.781738 2302 raft_consensus.cc:3037] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:26.785028 2441 raft_consensus.cc:2454] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 1.
I20260923 21:00:26.785226 2302 raft_consensus.cc:2454] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 1.
I20260923 21:00:26.785667 2511 leader_election.cc:302] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, d247ea347bbb47a3a6ba3bfe1c42e9ea; no voters:
I20260923 21:00:26.786046 2655 raft_consensus.cc:2781] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:26.789062 2675 leader_election.cc:288] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 election: Requested vote from peers e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:26.789948 2655 raft_consensus.cc:686] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 1 LEADER]: Becoming Leader. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Running, Role: LEADER
I20260923 21:00:26.790271 2655 consensus_queue.cc:237] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:26.794935 2143 catalog_manager.cc:5495] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b reported cstate change: term changed from 0 to 1, leader changed from <none> to 32808b1db26c45a195b4e1f56517e20b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } health_report { overall_health: HEALTHY } } }
I20260923 21:00:26.844826 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.837091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.0.97.3:35165)
0923 21:00:26.837302 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.837308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.837326 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:00:26.837395 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.837400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.837483 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.837575 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:26.838138 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.839510 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.842030 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.842813 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.842941 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.843566 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.843729 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.843806 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.844423 (+ 617us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.844508 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:00:26.872185 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.865807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35165 (local address 127.0.97.3:43279)
0923 21:00:26.866014 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:26.866034 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:26.866200 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:26.866590 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:26.866595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:26.866608 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:00:26.866955 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.866966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.868525 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.868530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.869955 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:26.869970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.871029 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.871034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:26.871160 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.871200 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.871632 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.871641 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 21:00:26.871711 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 21:00:26.871915 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 21:00:26.872221 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:26.865868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43279 (local address 127.0.97.3:35165)
0923 21:00:26.866011 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:26.866017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:26.866119 (+ 102us) server_negotiation.cc:407] Connection header received
0923 21:00:26.866323 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:26.866353 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:26.866454 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:26.866548 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:26.867108 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.868329 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.870130 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:26.870841 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:26.870987 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:26.871385 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:26.871527 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:26.871622 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:26.871761 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 21:00:26.871851 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":53}
I20260923 21:00:26.882184 2441 raft_consensus.cc:1261] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Refusing update from remote peer 32808b1db26c45a195b4e1f56517e20b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:26.882405 2302 raft_consensus.cc:1261] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Refusing update from remote peer 32808b1db26c45a195b4e1f56517e20b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:26.882970 2706 consensus_queue.cc:1035] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:26.883416 2655 consensus_queue.cc:1035] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 21:00:26.906880 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 21:00:28.506500 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:28.506809 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:28.511287 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Sending DeleteTablet for 3 replicas of tablet f4a622d6a6d94e9281e8a15bbd51a2d2
I20260923 21:00:28.512327 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a622d6a6d94e9281e8a15bbd51a2d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55818
I20260923 21:00:28.512398 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a622d6a6d94e9281e8a15bbd51a2d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46056
I20260923 21:00:28.512391 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a622d6a6d94e9281e8a15bbd51a2d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40930
I20260923 21:00:28.528190 2733 tablet_replica.cc:330] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: stopping tablet replica
I20260923 21:00:28.528618 2733 raft_consensus.cc:2229] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:28.529038 2733 raft_consensus.cc:2258] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:28.530596 2733 ts_tablet_manager.cc:1905] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:28.536206 2732 tablet_replica.cc:330] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: stopping tablet replica
I20260923 21:00:28.536478 2734 tablet_replica.cc:330] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: stopping tablet replica
I20260923 21:00:28.536504 2732 raft_consensus.cc:2229] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:28.536857 2732 raft_consensus.cc:2258] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:28.536922 2734 raft_consensus.cc:2229] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:28.537549 2734 raft_consensus.cc:2258] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:28.538626 2734 ts_tablet_manager.cc:1905] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:28.539038 2732 ts_tablet_manager.cc:1905] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:28.554078 2733 ts_tablet_manager.cc:1918] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 21:00:28.554368 2733 log.cc:1196] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/f4a622d6a6d94e9281e8a15bbd51a2d2
I20260923 21:00:28.554898 2733 ts_tablet_manager.cc:1939] T f4a622d6a6d94e9281e8a15bbd51a2d2 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting consensus metadata
Sep 23 21:00:28 dist-test-slave-8xn5 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260923 21:00:28.617636 2130 catalog_manager.cc:4841] TS e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979): tablet f4a622d6a6d94e9281e8a15bbd51a2d2 (table test-table [id=ebed19d64c0c43d59a13519f5ebea1da]) successfully deleted
W20260923 21:00:28.642674 388 mini_kdc.cc:317] Time spent kinit for test-user: real 0.128s user 0.000s sys 0.012s
I20260923 21:00:28.642841 2734 ts_tablet_manager.cc:1918] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 21:00:28.642879 2732 ts_tablet_manager.cc:1918] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 21:00:28.643007 2734 log.cc:1196] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/f4a622d6a6d94e9281e8a15bbd51a2d2
I20260923 21:00:28.643039 2732 log.cc:1196] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/f4a622d6a6d94e9281e8a15bbd51a2d2
I20260923 21:00:28.643527 2732 ts_tablet_manager.cc:1939] T f4a622d6a6d94e9281e8a15bbd51a2d2 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting consensus metadata
I20260923 21:00:28.643545 2734 ts_tablet_manager.cc:1939] T f4a622d6a6d94e9281e8a15bbd51a2d2 P 32808b1db26c45a195b4e1f56517e20b: Deleting consensus metadata
I20260923 21:00:28.645077 2132 catalog_manager.cc:4841] TS 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165): tablet f4a622d6a6d94e9281e8a15bbd51a2d2 (table test-table [id=ebed19d64c0c43d59a13519f5ebea1da]) successfully deleted
I20260923 21:00:28.645282 2129 catalog_manager.cc:4841] TS d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847): tablet f4a622d6a6d94e9281e8a15bbd51a2d2 (table test-table [id=ebed19d64c0c43d59a13519f5ebea1da]) successfully deleted
Sep 23 21:00:28 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:28.664304 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:28.647597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.0.97.62:37889)
0923 21:00:28.648244 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:28.648248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:28.648517 (+ 269us) server_negotiation.cc:407] Connection header received
0923 21:00:28.648808 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:28.648812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:28.648955 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:28.649109 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:28.650779 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:28.651610 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:28.652993 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:28.653415 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:28.653551 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:28.657864 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:28.657872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:28.657875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:28.657920 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:28.661054 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:28.661815 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:28.661820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:28.661823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:28.661966 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:28.662461 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:28.662465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:28.662467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:28.662951 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:28.663146 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:28.663475 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 21:00:28.663771 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":379,"threads_started":1}
I20260923 21:00:28.668371 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:28.668975 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:28.675858 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 1896c4135c6e4d6cada4b36b06843064 (DEFAULT_TABLE table=test-table [id=7c25d5e8f1914b26bf934f4f14da491b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:28.676004 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 1896c4135c6e4d6cada4b36b06843064 (DEFAULT_TABLE table=test-table [id=7c25d5e8f1914b26bf934f4f14da491b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:28.676095 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 1896c4135c6e4d6cada4b36b06843064 (DEFAULT_TABLE table=test-table [id=7c25d5e8f1914b26bf934f4f14da491b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:28.676340 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1896c4135c6e4d6cada4b36b06843064. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:28.676478 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1896c4135c6e4d6cada4b36b06843064. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:28.676478 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1896c4135c6e4d6cada4b36b06843064. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:28.683612 2743 tablet_bootstrap.cc:492] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Bootstrap starting.
I20260923 21:00:28.684641 2745 tablet_bootstrap.cc:492] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Bootstrap starting.
I20260923 21:00:28.685319 2744 tablet_bootstrap.cc:492] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: Bootstrap starting.
I20260923 21:00:28.685892 2743 tablet_bootstrap.cc:654] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:28.687130 2744 tablet_bootstrap.cc:654] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:28.687353 2745 tablet_bootstrap.cc:654] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:28.689477 2744 tablet_bootstrap.cc:492] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: No bootstrap required, opened a new log
I20260923 21:00:28.690042 2744 ts_tablet_manager.cc:1397] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 21:00:28.690191 2745 tablet_bootstrap.cc:492] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: No bootstrap required, opened a new log
I20260923 21:00:28.690366 2745 ts_tablet_manager.cc:1397] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 21:00:28.691035 2745 raft_consensus.cc:348] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.690985 2744 raft_consensus.cc:348] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.691238 2745 raft_consensus.cc:374] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:28.691244 2744 raft_consensus.cc:374] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:28.691337 2745 raft_consensus.cc:729] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247ea347bbb47a3a6ba3bfe1c42e9ea, State: Initialized, Role: FOLLOWER
I20260923 21:00:28.691359 2744 raft_consensus.cc:729] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Initialized, Role: FOLLOWER
I20260923 21:00:28.691480 2743 tablet_bootstrap.cc:492] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: No bootstrap required, opened a new log
I20260923 21:00:28.691627 2743 ts_tablet_manager.cc:1397] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 21:00:28.691579 2745 consensus_queue.cc:260] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.691619 2744 consensus_queue.cc:260] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.692412 2745 ts_tablet_manager.cc:1428] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 21:00:28.692459 2743 raft_consensus.cc:348] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.692754 2743 raft_consensus.cc:374] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:28.692871 2743 raft_consensus.cc:729] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ee0b4e519b4dc9b2b8aa0850646a1d, State: Initialized, Role: FOLLOWER
I20260923 21:00:28.693069 2744 ts_tablet_manager.cc:1428] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 21:00:28.693137 2743 consensus_queue.cc:260] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.694100 2743 ts_tablet_manager.cc:1428] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 21:00:28.779544 2750 raft_consensus.cc:482] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:28.779865 2750 raft_consensus.cc:504] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.780855 2750 leader_election.cc:288] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847), 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165)
I20260923 21:00:28.781406 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1896c4135c6e4d6cada4b36b06843064" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" is_pre_election: true
I20260923 21:00:28.781410 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1896c4135c6e4d6cada4b36b06843064" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32808b1db26c45a195b4e1f56517e20b" is_pre_election: true
I20260923 21:00:28.781682 2440 raft_consensus.cc:2454] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 0.
I20260923 21:00:28.781818 2580 raft_consensus.cc:2454] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 0.
I20260923 21:00:28.782251 2232 leader_election.cc:302] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d247ea347bbb47a3a6ba3bfe1c42e9ea, e0ee0b4e519b4dc9b2b8aa0850646a1d; no voters:
I20260923 21:00:28.782647 2750 raft_consensus.cc:2781] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:28.782750 2750 raft_consensus.cc:482] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:28.782856 2750 raft_consensus.cc:3037] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:28.785156 2750 raft_consensus.cc:504] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.785761 2750 leader_election.cc:288] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 election: Requested vote from peers d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847), 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165)
I20260923 21:00:28.786489 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1896c4135c6e4d6cada4b36b06843064" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
I20260923 21:00:28.786471 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1896c4135c6e4d6cada4b36b06843064" candidate_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32808b1db26c45a195b4e1f56517e20b"
I20260923 21:00:28.786703 2440 raft_consensus.cc:3037] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:28.786713 2580 raft_consensus.cc:3037] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:28.789743 2580 raft_consensus.cc:2454] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 1.
I20260923 21:00:28.789729 2440 raft_consensus.cc:2454] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0ee0b4e519b4dc9b2b8aa0850646a1d in term 1.
I20260923 21:00:28.790254 2232 leader_election.cc:302] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, d247ea347bbb47a3a6ba3bfe1c42e9ea, e0ee0b4e519b4dc9b2b8aa0850646a1d; no voters:
I20260923 21:00:28.790680 2750 raft_consensus.cc:2781] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:28.791138 2750 raft_consensus.cc:686] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 LEADER]: Becoming Leader. State: Replica: e0ee0b4e519b4dc9b2b8aa0850646a1d, State: Running, Role: LEADER
I20260923 21:00:28.791460 2750 consensus_queue.cc:237] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.794087 2145 catalog_manager.cc:5495] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } health_report { overall_health: UNKNOWN } } }
I20260923 21:00:28.824376 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:28.818155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.0.97.1:37979)
0923 21:00:28.818925 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:28.818929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:28.818949 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:00:28.819003 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:28.819007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:28.819066 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:28.819153 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:28.819573 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:28.820563 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:28.822583 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:28.823006 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:28.823115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:28.823594 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:28.823697 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:28.823777 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:28.823926 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 21:00:28.823991 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":494,"threads_started":1}
I20260923 21:00:28.834520 2440 raft_consensus.cc:1261] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Refusing update from remote peer e0ee0b4e519b4dc9b2b8aa0850646a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:28.834558 2580 raft_consensus.cc:1261] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Refusing update from remote peer e0ee0b4e519b4dc9b2b8aa0850646a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:28.835048 2751 consensus_queue.cc:1035] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:28.835405 2750 consensus_queue.cc:1035] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:28.887871 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:28.888132 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:28.891439 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Sending DeleteTablet for 3 replicas of tablet 1896c4135c6e4d6cada4b36b06843064
I20260923 21:00:28.892315 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 1896c4135c6e4d6cada4b36b06843064 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46056
I20260923 21:00:28.892354 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet 1896c4135c6e4d6cada4b36b06843064 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55818
I20260923 21:00:28.892488 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 1896c4135c6e4d6cada4b36b06843064 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40930
I20260923 21:00:28.892700 2733 tablet_replica.cc:330] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: stopping tablet replica
I20260923 21:00:28.892911 2733 raft_consensus.cc:2229] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:28.893231 2734 tablet_replica.cc:330] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: stopping tablet replica
I20260923 21:00:28.893246 2733 raft_consensus.cc:2258] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:28.893448 2734 raft_consensus.cc:2229] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:28.893644 2734 raft_consensus.cc:2258] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:28.893952 2732 tablet_replica.cc:330] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: stopping tablet replica
I20260923 21:00:28.894155 2732 raft_consensus.cc:2229] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:28.894402 2732 raft_consensus.cc:2258] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:28.894726 2733 ts_tablet_manager.cc:1905] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:28.894757 2734 ts_tablet_manager.cc:1905] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:28.893769 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:28.895236 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:28.897073 2732 ts_tablet_manager.cc:1905] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:28.900604 2734 ts_tablet_manager.cc:1918] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:28.900755 2734 log.cc:1196] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/1896c4135c6e4d6cada4b36b06843064
I20260923 21:00:28.901170 2734 ts_tablet_manager.cc:1939] T 1896c4135c6e4d6cada4b36b06843064 P 32808b1db26c45a195b4e1f56517e20b: Deleting consensus metadata
I20260923 21:00:28.901525 2733 ts_tablet_manager.cc:1918] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:28.901659 2733 log.cc:1196] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/1896c4135c6e4d6cada4b36b06843064
I20260923 21:00:28.902257 2132 catalog_manager.cc:4841] TS 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165): tablet 1896c4135c6e4d6cada4b36b06843064 (table test-table [id=7c25d5e8f1914b26bf934f4f14da491b]) successfully deleted
I20260923 21:00:28.902992 2282 tablet_service.cc:1467] Processing CreateTablet for tablet f9988440ea5341198040c396d90cb637 (DEFAULT_TABLE table=test-table [id=9b8ba750c375445e8bc7a797be69e527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:28.903128 2421 tablet_service.cc:1467] Processing CreateTablet for tablet f9988440ea5341198040c396d90cb637 (DEFAULT_TABLE table=test-table [id=9b8ba750c375445e8bc7a797be69e527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:28.903439 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9988440ea5341198040c396d90cb637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:28.903611 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9988440ea5341198040c396d90cb637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:28.904297 2732 ts_tablet_manager.cc:1918] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:00:28.904464 2732 log.cc:1196] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/1896c4135c6e4d6cada4b36b06843064
I20260923 21:00:28.904946 2732 ts_tablet_manager.cc:1939] T 1896c4135c6e4d6cada4b36b06843064 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting consensus metadata
I20260923 21:00:28.906165 2129 catalog_manager.cc:4841] TS d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847): tablet 1896c4135c6e4d6cada4b36b06843064 (table test-table [id=7c25d5e8f1914b26bf934f4f14da491b]) successfully deleted
I20260923 21:00:28.906230 2733 ts_tablet_manager.cc:1939] T 1896c4135c6e4d6cada4b36b06843064 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting consensus metadata
I20260923 21:00:28.907424 2130 catalog_manager.cc:4841] TS e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979): tablet 1896c4135c6e4d6cada4b36b06843064 (table test-table [id=7c25d5e8f1914b26bf934f4f14da491b]) successfully deleted
I20260923 21:00:28.907837 2560 tablet_service.cc:1467] Processing CreateTablet for tablet f9988440ea5341198040c396d90cb637 (DEFAULT_TABLE table=test-table [id=9b8ba750c375445e8bc7a797be69e527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:28.908293 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9988440ea5341198040c396d90cb637. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:28.908901 2745 tablet_bootstrap.cc:492] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Bootstrap starting.
I20260923 21:00:28.909511 2743 tablet_bootstrap.cc:492] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Bootstrap starting.
I20260923 21:00:28.910545 2745 tablet_bootstrap.cc:654] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:28.911049 2743 tablet_bootstrap.cc:654] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:28.912799 2745 tablet_bootstrap.cc:492] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: No bootstrap required, opened a new log
I20260923 21:00:28.912981 2745 ts_tablet_manager.cc:1397] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:28.913712 2745 raft_consensus.cc:348] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.913952 2745 raft_consensus.cc:374] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:28.914023 2745 raft_consensus.cc:729] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247ea347bbb47a3a6ba3bfe1c42e9ea, State: Initialized, Role: FOLLOWER
I20260923 21:00:28.914700 2745 consensus_queue.cc:260] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.915362 2744 tablet_bootstrap.cc:492] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: Bootstrap starting.
I20260923 21:00:28.916144 2745 ts_tablet_manager.cc:1428] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:28.917059 2743 tablet_bootstrap.cc:492] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: No bootstrap required, opened a new log
I20260923 21:00:28.917181 2744 tablet_bootstrap.cc:654] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:28.917227 2743 ts_tablet_manager.cc:1397] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 21:00:28.918635 2743 raft_consensus.cc:348] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.918924 2743 raft_consensus.cc:374] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:28.919015 2743 raft_consensus.cc:729] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ee0b4e519b4dc9b2b8aa0850646a1d, State: Initialized, Role: FOLLOWER
I20260923 21:00:28.919263 2743 consensus_queue.cc:260] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.920024 2743 ts_tablet_manager.cc:1428] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:28.920125 2744 tablet_bootstrap.cc:492] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: No bootstrap required, opened a new log
I20260923 21:00:28.920322 2744 ts_tablet_manager.cc:1397] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 21:00:28.922123 2744 raft_consensus.cc:348] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.922456 2744 raft_consensus.cc:374] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:28.922545 2744 raft_consensus.cc:729] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Initialized, Role: FOLLOWER
I20260923 21:00:28.922807 2744 consensus_queue.cc:260] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.923305 2744 ts_tablet_manager.cc:1428] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 21:00:28.939237 2714 raft_consensus.cc:482] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:28.939492 2714 raft_consensus.cc:504] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.940233 2714 leader_election.cc:288] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:28.940896 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9988440ea5341198040c396d90cb637" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" is_pre_election: true
I20260923 21:00:28.940953 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9988440ea5341198040c396d90cb637" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" is_pre_election: true
I20260923 21:00:28.941175 2302 raft_consensus.cc:2454] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 0.
I20260923 21:00:28.941246 2440 raft_consensus.cc:2454] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 0.
I20260923 21:00:28.941676 2512 leader_election.cc:302] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, e0ee0b4e519b4dc9b2b8aa0850646a1d; no voters:
I20260923 21:00:28.942009 2714 raft_consensus.cc:2781] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:28.942163 2714 raft_consensus.cc:482] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:28.942250 2714 raft_consensus.cc:3037] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:28.944262 2714 raft_consensus.cc:504] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.944839 2714 leader_election.cc:288] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 election: Requested vote from peers e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979), d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847)
I20260923 21:00:28.945384 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9988440ea5341198040c396d90cb637" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d"
I20260923 21:00:28.945441 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9988440ea5341198040c396d90cb637" candidate_uuid: "32808b1db26c45a195b4e1f56517e20b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea"
I20260923 21:00:28.945616 2302 raft_consensus.cc:3037] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:28.945631 2440 raft_consensus.cc:3037] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:28.947753 2440 raft_consensus.cc:2454] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 1.
I20260923 21:00:28.947822 2302 raft_consensus.cc:2454] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32808b1db26c45a195b4e1f56517e20b in term 1.
I20260923 21:00:28.948208 2511 leader_election.cc:302] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32808b1db26c45a195b4e1f56517e20b, d247ea347bbb47a3a6ba3bfe1c42e9ea; no voters:
I20260923 21:00:28.948544 2714 raft_consensus.cc:2781] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:28.948693 2714 raft_consensus.cc:686] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 1 LEADER]: Becoming Leader. State: Replica: 32808b1db26c45a195b4e1f56517e20b, State: Running, Role: LEADER
I20260923 21:00:28.949004 2714 consensus_queue.cc:237] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } }
I20260923 21:00:28.951838 2144 catalog_manager.cc:5495] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b reported cstate change: term changed from 0 to 1, leader changed from <none> to 32808b1db26c45a195b4e1f56517e20b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "32808b1db26c45a195b4e1f56517e20b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32808b1db26c45a195b4e1f56517e20b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35165 } health_report { overall_health: HEALTHY } } }
I20260923 21:00:28.977118 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:28.971288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.97.3:35165)
0923 21:00:28.971894 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:28.971899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:28.971918 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:00:28.971970 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:28.971973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:28.972036 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:28.972119 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:28.972526 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:28.973868 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:28.975296 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:28.975811 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:28.975913 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:28.976294 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:28.976405 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:28.976472 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:28.976606 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 21:00:28.976784 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":361,"threads_started":1}
I20260923 21:00:28.986145 2440 raft_consensus.cc:1261] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Refusing update from remote peer 32808b1db26c45a195b4e1f56517e20b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:28.986394 2302 raft_consensus.cc:1261] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Refusing update from remote peer 32808b1db26c45a195b4e1f56517e20b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:28.986817 2727 consensus_queue.cc:1035] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d247ea347bbb47a3a6ba3bfe1c42e9ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:28.987329 2714 consensus_queue.cc:1035] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0ee0b4e519b4dc9b2b8aa0850646a1d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 21:00:29.008615 2712 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 21:00:30.596127 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:30.596400 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:30.600842 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e442478b33ca45bab91a7fbaf062d1ad: Sending DeleteTablet for 3 replicas of tablet f9988440ea5341198040c396d90cb637
I20260923 21:00:30.601804 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet f9988440ea5341198040c396d90cb637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46056
I20260923 21:00:30.601827 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet f9988440ea5341198040c396d90cb637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55818
I20260923 21:00:30.601919 2559 tablet_service.cc:1514] Processing DeleteTablet for tablet f9988440ea5341198040c396d90cb637 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40930
I20260923 21:00:30.602711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2216
I20260923 21:00:30.615939 2787 tablet_replica.cc:330] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: stopping tablet replica
I20260923 21:00:30.616448 2787 raft_consensus.cc:2229] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:30.616920 2787 raft_consensus.cc:2258] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:30.618289 2787 ts_tablet_manager.cc:1905] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:30.626886 2788 tablet_replica.cc:330] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: stopping tablet replica
Sep 23 21:00:30 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:30.627049 2789 tablet_replica.cc:330] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: stopping tablet replica
I20260923 21:00:30.627141 2788 raft_consensus.cc:2229] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:30.627259 2789 raft_consensus.cc:2229] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:30.627426 2788 raft_consensus.cc:2258] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:30.628043 2789 raft_consensus.cc:2258] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:30.628489 2788 ts_tablet_manager.cc:1905] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:30.628695 2787 ts_tablet_manager.cc:1918] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 21:00:30.628875 2787 log.cc:1196] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/f9988440ea5341198040c396d90cb637
I20260923 21:00:30.629465 2787 ts_tablet_manager.cc:1939] T f9988440ea5341198040c396d90cb637 P e0ee0b4e519b4dc9b2b8aa0850646a1d: Deleting consensus metadata
I20260923 21:00:30.629554 2789 ts_tablet_manager.cc:1905] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:30.630708 2130 catalog_manager.cc:4841] TS e0ee0b4e519b4dc9b2b8aa0850646a1d (127.0.97.1:37979): tablet f9988440ea5341198040c396d90cb637 (table test-table [id=9b8ba750c375445e8bc7a797be69e527]) successfully deleted
I20260923 21:00:30.634663 2788 ts_tablet_manager.cc:1918] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 21:00:30.634819 2788 log.cc:1196] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/f9988440ea5341198040c396d90cb637
I20260923 21:00:30.635314 2788 ts_tablet_manager.cc:1939] T f9988440ea5341198040c396d90cb637 P d247ea347bbb47a3a6ba3bfe1c42e9ea: Deleting consensus metadata
I20260923 21:00:30.636617 2129 catalog_manager.cc:4841] TS d247ea347bbb47a3a6ba3bfe1c42e9ea (127.0.97.2:37847): tablet f9988440ea5341198040c396d90cb637 (table test-table [id=9b8ba750c375445e8bc7a797be69e527]) successfully deleted
I20260923 21:00:30.637219 2789 ts_tablet_manager.cc:1918] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 21:00:30.637334 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:30.604246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.97.1:37979)
0923 21:00:30.612077 (+ 7831us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:30.612086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:30.612121 (+ 35us) server_negotiation.cc:407] Connection header received
0923 21:00:30.612212 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:30.612221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:30.612323 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:30.612463 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:30.615565 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:30.617443 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:30.620338 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:30.621284 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:30.621458 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:30.628792 (+ 7334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:30.628816 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:30.628841 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:30.628906 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:30.633092 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:30.634084 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:30.634099 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:30.634103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:30.634768 (+ 665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:30.635348 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:30.635354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:30.635358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:30.635847 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:30.636154 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:30.636664 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 21:00:30.636927 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7649,"thread_start_us":7513,"threads_started":1}
I20260923 21:00:30.637365 2789 log.cc:1196] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/f9988440ea5341198040c396d90cb637
I20260923 21:00:30.637877 2789 ts_tablet_manager.cc:1939] T f9988440ea5341198040c396d90cb637 P 32808b1db26c45a195b4e1f56517e20b: Deleting consensus metadata
W20260923 21:00:30.638093 2347 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46072
W20260923 21:00:30.638898 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:30.639034 2132 catalog_manager.cc:4841] TS 32808b1db26c45a195b4e1f56517e20b (127.0.97.3:35165): tablet f9988440ea5341198040c396d90cb637 (table test-table [id=9b8ba750c375445e8bc7a797be69e527]) successfully deleted
I20260923 21:00:30.639163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 2216
I20260923 21:00:30.662317 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2356
Sep 23 21:00:30 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:00:30.679643 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:30.663413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.0.97.2:37847)
0923 21:00:30.663939 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:30.663943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:30.663960 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:00:30.664013 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:30.664037 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:30.664104 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:30.664246 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:30.665577 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:30.666437 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:30.667692 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:30.668145 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:30.668271 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:30.673480 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:30.673497 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:30.673514 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:30.673558 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:30.676669 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:30.677445 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:30.677454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:30.677457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:30.677679 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:30.678121 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:30.678125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:30.678127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:30.678662 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:30.678907 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:30.679194 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 21:00:30.679379 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":282,"threads_started":1}
W20260923 21:00:30.680436 2486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55832
W20260923 21:00:30.681071 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2356: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:30.681272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 2356
I20260923 21:00:30.699373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2495
Sep 23 21:00:30 dist-test-slave-8xn5 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197228, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:00:30.715809 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:30.700328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.0.97.3:35165)
0923 21:00:30.700840 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:30.700845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:30.700860 (+ 15us) server_negotiation.cc:407] Connection header received
0923 21:00:30.700913 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:30.700917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:30.700999 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:30.701096 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:30.702508 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:30.703232 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:30.704431 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:30.704866 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:30.704966 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:30.709410 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:30.709427 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:30.709443 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:30.709481 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:30.712510 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:30.713367 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:30.713382 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:30.713385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:30.713547 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:30.713991 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:30.713994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:30.713997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:30.714483 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:30.714730 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:30.715125 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 21:00:30.715311 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":266,"threads_started":1}
W20260923 21:00:30.716614 2625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40982
W20260923 21:00:30.717298 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2495: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:30.717437 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 2495
I20260923 21:00:30.743234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2113
I20260923 21:00:30.744498 2185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:30.868621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 2113
2026-09-23T21:00:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9131 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:30 dist-test-slave-8xn5 krb5kdc[2797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:30 dist-test-slave-8xn5 krb5kdc[2797](info): set up 2 sockets
Sep 23 21:00:30 dist-test-slave-8xn5 krb5kdc[2797](info): commencing operation
krb5kdc: starting...
W20260923 21:00:32.034962 388 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:32 dist-test-slave-8xn5 krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:32.246042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39399
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:32.587699 2810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:32.588002 2810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:32.588091 2810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:32.595538 2810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:32.595657 2810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:32.595702 2810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:32.595759 2810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:32.595803 2810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:32.605862 2810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:32.607246 2810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:32.608660 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:32.620987 2815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:32.621021 2816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:32.620987 2818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:32.621979 2810 server_base.cc:1034] running on GCE node
I20260923 21:00:32.622946 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:32.624501 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:32.625779 2810 hybrid_clock.cc:648] HybridClock initialized: now 1790197232625735 us; error 38 us; skew 500 ppm
I20260923 21:00:32.626160 2810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:32 dist-test-slave-8xn5 krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:32.629577 2810 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:32.631387 2810 webserver.cc:460] Webserver started at http://127.0.97.62:41347/ using document root <none> and password file <none>
I20260923 21:00:32.632155 2810 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:32.632290 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:32.632650 2810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:32.636003 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "a19a52da09e749aba6d4f6c2361a75fc"
format_stamp: "Formatted at 2026-09-23 21:00:32 on dist-test-slave-8xn5"
server_key: "a10f99c20b9f1b3c343393502b92276e"
server_key_iv: "de825b17418981f811b94b720c39b908"
server_key_version: "encryptionkey@0"
I20260923 21:00:32.636878 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "a19a52da09e749aba6d4f6c2361a75fc"
format_stamp: "Formatted at 2026-09-23 21:00:32 on dist-test-slave-8xn5"
server_key: "a10f99c20b9f1b3c343393502b92276e"
server_key_iv: "de825b17418981f811b94b720c39b908"
server_key_version: "encryptionkey@0"
I20260923 21:00:32.642498 2810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 21:00:32.646157 2825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:32.647710 2810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:32.647898 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "a19a52da09e749aba6d4f6c2361a75fc"
format_stamp: "Formatted at 2026-09-23 21:00:32 on dist-test-slave-8xn5"
server_key: "a10f99c20b9f1b3c343393502b92276e"
server_key_iv: "de825b17418981f811b94b720c39b908"
server_key_version: "encryptionkey@0"
I20260923 21:00:32.648106 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:32.661897 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:32.666530 2810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:32.666812 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:32.685782 2810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39399
I20260923 21:00:32.685809 2876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39399 every 8 connection(s)
I20260923 21:00:32.687366 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:32.691540 2877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:32.697274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 2810
I20260923 21:00:32.697535 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:32.698457 388 external_mini_cluster.cc:1428] Setting key 8b25b3e821b531161e19b97a01b80d44
I20260923 21:00:32.705053 2877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc: Bootstrap starting.
I20260923 21:00:32.708680 2877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:32.709915 2877 log.cc:824] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:32.714088 2877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc: No bootstrap required, opened a new log
Sep 23 21:00:32 dist-test-slave-8xn5 krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:32.723742 2877 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a52da09e749aba6d4f6c2361a75fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260923 21:00:32.724191 2877 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:32.724337 2877 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19a52da09e749aba6d4f6c2361a75fc, State: Initialized, Role: FOLLOWER
I20260923 21:00:32.724900 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:32.700989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.97.62:39399)
0923 21:00:32.701919 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:32.701932 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:32.701977 (+ 45us) server_negotiation.cc:407] Connection header received
0923 21:00:32.703923 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:32.703950 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:32.704432 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:32.704850 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:32.706463 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:32.707840 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:32.709320 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:32.710042 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:32.710252 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:32.715904 (+ 5652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:32.715917 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:32.715933 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:32.715986 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:32.719939 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:32.720857 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:32.720866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:32.720883 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:32.721074 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:32.721547 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:32.721551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:32.721553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:32.722204 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:32.722634 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:32.723036 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 21:00:32.723535 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":416,"threads_started":1}
I20260923 21:00:32.725001 2877 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a52da09e749aba6d4f6c2361a75fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260923 21:00:32.725356 2877 raft_consensus.cc:388] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:32.725526 2877 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:32.725742 2877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:32.728698 2877 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a52da09e749aba6d4f6c2361a75fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260923 21:00:32.729185 2877 leader_election.cc:302] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a19a52da09e749aba6d4f6c2361a75fc; no voters:
I20260923 21:00:32.729759 2877 leader_election.cc:288] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:32.729915 2882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:32.730682 2882 raft_consensus.cc:686] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [term 1 LEADER]: Becoming Leader. State: Replica: a19a52da09e749aba6d4f6c2361a75fc, State: Running, Role: LEADER
I20260923 21:00:32.731303 2882 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a52da09e749aba6d4f6c2361a75fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } }
I20260923 21:00:32.731986 2877 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:32.734493 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19a52da09e749aba6d4f6c2361a75fc. Latest consensus state: current_term: 1 leader_uuid: "a19a52da09e749aba6d4f6c2361a75fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a52da09e749aba6d4f6c2361a75fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } } }
I20260923 21:00:32.734314 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a19a52da09e749aba6d4f6c2361a75fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a52da09e749aba6d4f6c2361a75fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39399 } } }
I20260923 21:00:32.734843 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:32.734874 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:32.737106 2888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:32.743119 2888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:32.752641 2888 catalog_manager.cc:1347] Generated new cluster ID: c6c2eb1a327c479e85197266e773f79c
I20260923 21:00:32.752812 2888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:32.785933 2888 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:32.787931 2888 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:32.798854 2888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a19a52da09e749aba6d4f6c2361a75fc: Generated new TSK 0
I20260923 21:00:32.799932 2888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:32.854744 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:33.236436 2903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:33.236737 2903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:33.236836 2903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:33.244450 2903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:33.244573 2903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:33.244688 2903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:33.255162 2903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:33.256544 2903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:33.258003 2903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:33.271354 2908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:33.272269 2909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:33.273614 2911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:33.274045 2903 server_base.cc:1034] running on GCE node
I20260923 21:00:33.274801 2903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:33.275987 2903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:33.277233 2903 hybrid_clock.cc:648] HybridClock initialized: now 1790197233277181 us; error 50 us; skew 500 ppm
I20260923 21:00:33.277571 2903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:33 dist-test-slave-8xn5 krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:33.281215 2903 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:33.282968 2903 webserver.cc:460] Webserver started at http://127.0.97.1:35305/ using document root <none> and password file <none>
I20260923 21:00:33.283742 2903 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:33.283892 2903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:33.284226 2903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:33.287451 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "823496082e75461896a0c4532e1554fa"
format_stamp: "Formatted at 2026-09-23 21:00:33 on dist-test-slave-8xn5"
server_key: "4d3169c1171655ff21946f02bbd93139"
server_key_iv: "6ccb2bfa0535ffc4525aaef49291a9c9"
server_key_version: "encryptionkey@0"
I20260923 21:00:33.288295 2903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "823496082e75461896a0c4532e1554fa"
format_stamp: "Formatted at 2026-09-23 21:00:33 on dist-test-slave-8xn5"
server_key: "4d3169c1171655ff21946f02bbd93139"
server_key_iv: "6ccb2bfa0535ffc4525aaef49291a9c9"
server_key_version: "encryptionkey@0"
I20260923 21:00:33.294080 2903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:00:33.298200 2918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:33.299880 2903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:33.300027 2903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "823496082e75461896a0c4532e1554fa"
format_stamp: "Formatted at 2026-09-23 21:00:33 on dist-test-slave-8xn5"
server_key: "4d3169c1171655ff21946f02bbd93139"
server_key_iv: "6ccb2bfa0535ffc4525aaef49291a9c9"
server_key_version: "encryptionkey@0"
I20260923 21:00:33.300215 2903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:33.335760 2903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:33.340627 2903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:33.340929 2903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:33.341922 2903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:33.343490 2903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:33.343566 2903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:33.343663 2903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:33.343716 2903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:33.379989 2903 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45957
I20260923 21:00:33.380093 3030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45957 every 8 connection(s)
I20260923 21:00:33.381397 2903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:33.387712 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 2903
I20260923 21:00:33.387966 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:33.388784 388 external_mini_cluster.cc:1428] Setting key 671b43eb3d3c7fd50bbe452891f31b13
Sep 23 21:00:33 dist-test-slave-8xn5 krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:33.407855 3033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:33.384477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35259 (local address 127.0.97.62:39399)
0923 21:00:33.384988 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:33.384993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:33.386186 (+ 1193us) server_negotiation.cc:407] Connection header received
0923 21:00:33.388805 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:33.388818 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:33.388929 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:33.389039 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:33.391011 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.391748 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.394551 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.394990 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.395101 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:33.399239 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:33.399248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:33.399256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:33.399304 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:33.403210 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:33.404389 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:33.404399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:33.404405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:33.404567 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:33.405253 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:33.405259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:33.405263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:33.405690 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:33.405874 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:33.406836 (+ 962us) server_negotiation.cc:299] Negotiation successful
0923 21:00:33.407117 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":280,"threads_started":1}
I20260923 21:00:33.408901 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:33.385149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39399 (local address 127.0.97.1:35259)
0923 21:00:33.385996 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:33.386045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:33.388390 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:33.389361 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:33.389374 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:33.390031 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:33.390841 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:33.390862 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.391916 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.391924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:33.394321 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:33.394375 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.395290 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.395300 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:33.395670 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:33.396517 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:33.396561 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:33.398924 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:33.403434 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:33.403443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:33.403588 (+ 145us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:33.404213 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:33.404773 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:33.404779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:33.404784 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:33.405050 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:33.405850 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:33.405857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:33.406274 (+ 417us) client_negotiation.cc:769] Sending connection context
0923 21:00:33.406654 (+ 380us) client_negotiation.cc:240] Negotiation successful
0923 21:00:33.407312 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":377,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:00:33.411154 3031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39399
I20260923 21:00:33.411468 3031 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:33.412289 3031 heartbeater.cc:511] Master 127.0.97.62:39399 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 21:00:33.415203 2842 ts_manager.cc:194] Registered new tserver with Master: 823496082e75461896a0c4532e1554fa (127.0.97.1:45957)
I20260923 21:00:33.417656 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:33.444561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:33.798597 3037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:33.798939 3037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:33.799043 3037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:33.806816 3037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:33.806973 3037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:33.807181 3037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:33.817629 3037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:33.819126 3037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:33.820645 3037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:33.835500 3043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:33.835474 3042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:33.835625 3045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:33.836344 3037 server_base.cc:1034] running on GCE node
I20260923 21:00:33.837167 3037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:33.838261 3037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:33.839651 3037 hybrid_clock.cc:648] HybridClock initialized: now 1790197233839597 us; error 56 us; skew 500 ppm
I20260923 21:00:33.840044 3037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:33 dist-test-slave-8xn5 krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:33.843719 3037 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:33.845384 3037 webserver.cc:460] Webserver started at http://127.0.97.2:39057/ using document root <none> and password file <none>
I20260923 21:00:33.846174 3037 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:33.846393 3037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:33.846765 3037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:33.850033 3037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "d88f54bdba8047f49f7179026f159b10"
format_stamp: "Formatted at 2026-09-23 21:00:33 on dist-test-slave-8xn5"
server_key: "4c2ac82782db0211f9a6d1494f9a0884"
server_key_iv: "37d784bef812edd9bdf87d4eebcf8566"
server_key_version: "encryptionkey@0"
I20260923 21:00:33.850988 3037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d88f54bdba8047f49f7179026f159b10"
format_stamp: "Formatted at 2026-09-23 21:00:33 on dist-test-slave-8xn5"
server_key: "4c2ac82782db0211f9a6d1494f9a0884"
server_key_iv: "37d784bef812edd9bdf87d4eebcf8566"
server_key_version: "encryptionkey@0"
I20260923 21:00:33.856911 3037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 21:00:33.860968 3052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:33.862905 3037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 21:00:33.863124 3037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "d88f54bdba8047f49f7179026f159b10"
format_stamp: "Formatted at 2026-09-23 21:00:33 on dist-test-slave-8xn5"
server_key: "4c2ac82782db0211f9a6d1494f9a0884"
server_key_iv: "37d784bef812edd9bdf87d4eebcf8566"
server_key_version: "encryptionkey@0"
I20260923 21:00:33.863363 3037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:33.889871 3037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:33.895375 3037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:33.895712 3037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:33.896854 3037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:33.898559 3037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:33.898675 3037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:33.898834 3037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:33.898906 3037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:33.937376 3164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43245 every 8 connection(s)
I20260923 21:00:33.937371 3037 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43245
I20260923 21:00:33.939038 3037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:33.945792 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3037
I20260923 21:00:33.945993 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:33.946652 388 external_mini_cluster.cc:1428] Setting key 6600e20da8f1283bd38cfb6365b022ae
Sep 23 21:00:33 dist-test-slave-8xn5 krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:33.965306 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:33.942104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34403 (local address 127.0.97.62:39399)
0923 21:00:33.942636 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:33.942641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:33.944011 (+ 1370us) server_negotiation.cc:407] Connection header received
0923 21:00:33.946525 (+ 2514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:33.946535 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:33.946626 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:33.946776 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:33.948670 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.950613 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.953599 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.954136 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.954257 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:33.958871 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:33.958877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:33.958880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:33.958922 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:33.961828 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:33.962832 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:33.962843 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:33.962845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:33.962976 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:33.963457 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:33.963461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:33.963463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:33.963753 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:33.963916 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:33.964677 (+ 761us) server_negotiation.cc:299] Negotiation successful
0923 21:00:33.964913 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":315,"threads_started":1}
I20260923 21:00:33.966465 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:33.942808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39399 (local address 127.0.97.2:34403)
0923 21:00:33.943791 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:33.943836 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:33.946019 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:33.947102 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:33.947116 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:33.947779 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:33.948479 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:33.948500 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.950814 (+ 2314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.950823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:33.953361 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:33.953385 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:33.954438 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:33.954448 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:33.954850 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:33.955805 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:33.955832 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:33.958545 (+ 2713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:33.962045 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:33.962055 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:33.962082 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:33.962670 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:33.963096 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:33.963101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:33.963105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:33.963307 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:33.963898 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:33.963907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:33.964294 (+ 387us) client_negotiation.cc:769] Sending connection context
0923 21:00:33.964577 (+ 283us) client_negotiation.cc:240] Negotiation successful
0923 21:00:33.965183 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":442,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:00:33.968153 3165 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39399
I20260923 21:00:33.968626 3165 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:33.969460 3165 heartbeater.cc:511] Master 127.0.97.62:39399 requested a full tablet report, sending...
I20260923 21:00:33.971657 2842 ts_manager.cc:194] Registered new tserver with Master: d88f54bdba8047f49f7179026f159b10 (127.0.97.2:43245)
I20260923 21:00:33.972591 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
W20260923 21:00:34.270763 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.3: real 0.321s user 0.000s sys 0.010s
I20260923 21:00:34.271591 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:00:34.421550 3031 heartbeater.cc:503] Master 127.0.97.62:39399 was elected leader, sending a full tablet report...
W20260923 21:00:34.609737 3171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:34.610044 3171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:34.610138 3171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:34.617748 3171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:34.617882 3171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:34.617996 3171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:34.628449 3171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39399
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:34.629854 3171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:34.631263 3171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:34.646286 3176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:34.646427 3177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:34.646397 3179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:34.647280 3171 server_base.cc:1034] running on GCE node
I20260923 21:00:34.648042 3171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:34.649250 3171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:34.650477 3171 hybrid_clock.cc:648] HybridClock initialized: now 1790197234650412 us; error 65 us; skew 500 ppm
I20260923 21:00:34.650875 3171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:34 dist-test-slave-8xn5 krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:34.654627 3171 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:34.656317 3171 webserver.cc:460] Webserver started at http://127.0.97.3:36537/ using document root <none> and password file <none>
I20260923 21:00:34.657089 3171 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:34.657233 3171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:34.657621 3171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:34.661324 3171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "0fc464a7fb6546e8a16a6b2bb7594a94"
format_stamp: "Formatted at 2026-09-23 21:00:34 on dist-test-slave-8xn5"
server_key: "ddf500e7135d4b26c1b3097f6927ce7a"
server_key_iv: "15dc18572e3322fbc5bfae21e2652435"
server_key_version: "encryptionkey@0"
I20260923 21:00:34.662257 3171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0fc464a7fb6546e8a16a6b2bb7594a94"
format_stamp: "Formatted at 2026-09-23 21:00:34 on dist-test-slave-8xn5"
server_key: "ddf500e7135d4b26c1b3097f6927ce7a"
server_key_iv: "15dc18572e3322fbc5bfae21e2652435"
server_key_version: "encryptionkey@0"
I20260923 21:00:34.668148 3171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 21:00:34.672377 3186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:34.674098 3171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:00:34.674256 3171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "0fc464a7fb6546e8a16a6b2bb7594a94"
format_stamp: "Formatted at 2026-09-23 21:00:34 on dist-test-slave-8xn5"
server_key: "ddf500e7135d4b26c1b3097f6927ce7a"
server_key_iv: "15dc18572e3322fbc5bfae21e2652435"
server_key_version: "encryptionkey@0"
I20260923 21:00:34.674539 3171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:34.687382 3171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:34.693310 3171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:34.693569 3171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:34.694803 3171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:34.696287 3171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:34.696383 3171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:34.696477 3171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:34.696513 3171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:34.728684 3171 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37065
I20260923 21:00:34.728724 3298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37065 every 8 connection(s)
I20260923 21:00:34.729985 3171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:34.731177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3171
I20260923 21:00:34.731374 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:34.731873 388 external_mini_cluster.cc:1428] Setting key f7df2acd3977610ceb992355430de450
Sep 23 21:00:34 dist-test-slave-8xn5 krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:34.751909 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:34.733370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49601 (local address 127.0.97.62:39399)
0923 21:00:34.733959 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:34.733965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:34.735173 (+ 1208us) server_negotiation.cc:407] Connection header received
0923 21:00:34.737079 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:34.737085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:34.737182 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:34.737272 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:34.738856 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:34.739599 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:34.741168 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:34.741562 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:34.741673 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:34.745466 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:34.745473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:34.745478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:34.745521 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:34.748807 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:34.749583 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:34.749587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:34.749589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:34.749750 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:34.750187 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:34.750190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:34.750192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:34.750521 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:34.750667 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:34.751414 (+ 747us) server_negotiation.cc:299] Negotiation successful
0923 21:00:34.751587 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":331,"threads_started":1}
I20260923 21:00:34.752918 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:34.734095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39399 (local address 127.0.97.3:49601)
0923 21:00:34.734967 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:34.735032 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:34.736779 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:34.737477 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:34.737486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:34.738011 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:34.738699 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:34.738717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:34.739765 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:34.739769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:34.741004 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:34.741021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:34.741803 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:34.741808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:34.742004 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:34.742789 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:34.742813 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:34.745156 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:34.748959 (+ 3803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:34.748967 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:34.748982 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:34.749467 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:34.749853 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:34.749856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:34.749859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:34.750049 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:34.750630 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:34.750636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:34.751004 (+ 368us) client_negotiation.cc:769] Sending connection context
0923 21:00:34.751283 (+ 279us) client_negotiation.cc:240] Negotiation successful
0923 21:00:34.751712 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":346,"threads_started":1}
I20260923 21:00:34.754804 3299 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39399
I20260923 21:00:34.755213 3299 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:34.756009 3299 heartbeater.cc:511] Master 127.0.97.62:39399 requested a full tablet report, sending...
I20260923 21:00:34.757731 2842 ts_manager.cc:194] Registered new tserver with Master: 0fc464a7fb6546e8a16a6b2bb7594a94 (127.0.97.3:37065)
I20260923 21:00:34.758733 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49601
I20260923 21:00:34.760787 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 21:00:34.794109 3301 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:00:34.791401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.0.97.62:39399)
0923 21:00:34.791708 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:34.791712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:34.792099 (+ 387us) server_negotiation.cc:407] Connection header received
0923 21:00:34.792261 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:34.792265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:34.792341 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:34.792451 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:34.793906 (+ 1455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54254: BlockingRecv error: recv got EOF from 127.0.0.1:54254 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
I20260923 21:00:34.797436 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2903
W20260923 21:00:34.800680 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2903: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:45957: server requires authentication, but client does not have Kerberos credentials available
I20260923 21:00:34.800889 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 2903
I20260923 21:00:34.816967 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3037
W20260923 21:00:34.819993 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3037: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43245: server requires authentication, but client does not have Kerberos credentials available
I20260923 21:00:34.820187 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3037
I20260923 21:00:34.835932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3171
W20260923 21:00:34.839277 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3171: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:37065: server requires authentication, but client does not have Kerberos credentials available
I20260923 21:00:34.839445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3171
I20260923 21:00:34.855670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 2810
I20260923 21:00:34.857016 2872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:34.960210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 2810
2026-09-23T21:00:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4085 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:35 dist-test-slave-8xn5 krb5kdc[3317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:35 dist-test-slave-8xn5 krb5kdc[3317](info): set up 2 sockets
Sep 23 21:00:35 dist-test-slave-8xn5 krb5kdc[3317](info): commencing operation
krb5kdc: starting...
W20260923 21:00:36.078723 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:36 dist-test-slave-8xn5 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:36.291662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41219
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:36.643508 3330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:36.643884 3330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:36.644040 3330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:36.651700 3330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:36.651865 3330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:36.651975 3330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:36.652096 3330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:36.652180 3330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:36.662760 3330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:36.664149 3330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:36.665603 3330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:36.678633 3335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:36.678738 3338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:36.678686 3336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:36.679956 3330 server_base.cc:1034] running on GCE node
I20260923 21:00:36.680770 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:36.682511 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:36.683745 3330 hybrid_clock.cc:648] HybridClock initialized: now 1790197236683671 us; error 66 us; skew 500 ppm
I20260923 21:00:36.684183 3330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:36 dist-test-slave-8xn5 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:36.693809 3330 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:36.695570 3330 webserver.cc:460] Webserver started at http://127.0.97.62:34279/ using document root <none> and password file <none>
I20260923 21:00:36.696328 3330 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:36.696445 3330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:36.696724 3330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:36.700068 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5"
format_stamp: "Formatted at 2026-09-23 21:00:36 on dist-test-slave-8xn5"
server_key: "676d7c371b8cd7e88c49d91e28adde9b"
server_key_iv: "f2eb929786cb0693c299689a08f02d49"
server_key_version: "encryptionkey@0"
I20260923 21:00:36.700944 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5"
format_stamp: "Formatted at 2026-09-23 21:00:36 on dist-test-slave-8xn5"
server_key: "676d7c371b8cd7e88c49d91e28adde9b"
server_key_iv: "f2eb929786cb0693c299689a08f02d49"
server_key_version: "encryptionkey@0"
I20260923 21:00:36.707238 3330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260923 21:00:36.711078 3345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:36.712769 3330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:00:36.712985 3330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5"
format_stamp: "Formatted at 2026-09-23 21:00:36 on dist-test-slave-8xn5"
server_key: "676d7c371b8cd7e88c49d91e28adde9b"
server_key_iv: "f2eb929786cb0693c299689a08f02d49"
server_key_version: "encryptionkey@0"
I20260923 21:00:36.713222 3330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:36.727275 3330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:36.732151 3330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:36.732470 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:36.752440 3330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41219
I20260923 21:00:36.752458 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41219 every 8 connection(s)
I20260923 21:00:36.753978 3330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:36.758085 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:36.763339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3330
I20260923 21:00:36.763571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:36.764376 388 external_mini_cluster.cc:1428] Setting key 4d47561d31a6fdc2a663f3340287f4b1
I20260923 21:00:36.770802 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5: Bootstrap starting.
I20260923 21:00:36.774748 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:36.775990 3397 log.cc:824] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:36.779731 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5: No bootstrap required, opened a new log
Sep 23 21:00:36 dist-test-slave-8xn5 krb5kdc[3317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:36.788813 3397 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260923 21:00:36.789256 3397 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:36.789409 3397 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1ffe54e7a74b45be69ca4c8b52e5f5, State: Initialized, Role: FOLLOWER
I20260923 21:00:36.790148 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:36.766540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.0.97.62:41219)
0923 21:00:36.767412 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:36.767426 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:36.767476 (+ 50us) server_negotiation.cc:407] Connection header received
0923 21:00:36.769327 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:36.769353 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:36.769794 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:36.770157 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:36.772064 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:36.773426 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:36.775244 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:36.775973 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:36.776127 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:36.781101 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:36.781110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:36.781128 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:36.781166 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:36.784841 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:36.785935 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:36.785945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:36.785952 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:36.786150 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:36.786644 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:36.786649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:36.786652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:36.787319 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:36.787707 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:36.788272 (+ 565us) server_negotiation.cc:299] Negotiation successful
0923 21:00:36.788769 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":397,"threads_started":1}
I20260923 21:00:36.790047 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260923 21:00:36.790439 3397 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:36.790627 3397 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:36.790830 3397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:36.793491 3397 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260923 21:00:36.794047 3397 leader_election.cc:302] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff1ffe54e7a74b45be69ca4c8b52e5f5; no voters:
I20260923 21:00:36.794394 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:00:36.794881 3397 leader_election.cc:288] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:36.794986 3402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:36.797591 3397 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:36.801957 3402 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [term 1 LEADER]: Becoming Leader. State: Replica: ff1ffe54e7a74b45be69ca4c8b52e5f5, State: Running, Role: LEADER
I20260923 21:00:36.802728 3402 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } }
I20260923 21:00:36.806030 3403 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } } }
I20260923 21:00:36.806098 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1ffe54e7a74b45be69ca4c8b52e5f5. Latest consensus state: current_term: 1 leader_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff1ffe54e7a74b45be69ca4c8b52e5f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41219 } } }
I20260923 21:00:36.806489 3403 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:36.806489 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:36.811661 3413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:36.817417 3413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:36.832787 3413 catalog_manager.cc:1347] Generated new cluster ID: ed9aa3dbb3a3480ab2e5ea8606ab2c39
I20260923 21:00:36.832935 3413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:36.854377 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:36.844793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.0.97.62:41219)
0923 21:00:36.845009 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:36.845016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:36.845655 (+ 639us) server_negotiation.cc:407] Connection header received
0923 21:00:36.845869 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:36.845881 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:36.845961 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:36.846074 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:36.847500 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:36.848556 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:36.850507 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:36.851138 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:36.851264 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:36.852607 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:36.852614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:36.852619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:36.853479 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:36.853564 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:36.853603 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:36.853714 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:36.853880 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 21:00:36.853985 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 21:00:36.858959 3413 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:36.860886 3413 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:36.878966 3413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff1ffe54e7a74b45be69ca4c8b52e5f5: Generated new TSK 0
I20260923 21:00:36.880178 3413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:00:36.890447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3330
I20260923 21:00:36.891844 3392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:36.984787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3330
2026-09-23T21:00:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2019 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:37 dist-test-slave-8xn5 krb5kdc[3431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:37 dist-test-slave-8xn5 krb5kdc[3431](info): set up 2 sockets
Sep 23 21:00:37 dist-test-slave-8xn5 krb5kdc[3431](info): commencing operation
krb5kdc: starting...
W20260923 21:00:38.098796 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:38 dist-test-slave-8xn5 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:38.311268 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41283
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:38.655342 3444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:38.655650 3444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:38.655858 3444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:38.663316 3444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:38.663457 3444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:38.663568 3444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:38.663667 3444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:38.663760 3444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:38.673951 3444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:38.675405 3444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:38.676854 3444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:38.689688 3449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:38.689610 3452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:38.689657 3450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:38.690889 3444 server_base.cc:1034] running on GCE node
I20260923 21:00:38.691849 3444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:38.693495 3444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:38.694777 3444 hybrid_clock.cc:648] HybridClock initialized: now 1790197238694731 us; error 40 us; skew 500 ppm
I20260923 21:00:38.695230 3444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:38 dist-test-slave-8xn5 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:38.698848 3444 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:38.700790 3444 webserver.cc:460] Webserver started at http://127.0.97.62:41023/ using document root <none> and password file <none>
I20260923 21:00:38.701622 3444 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:38.701784 3444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:38.702143 3444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:38.705569 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5"
format_stamp: "Formatted at 2026-09-23 21:00:38 on dist-test-slave-8xn5"
server_key: "55a278cc644d3afe59b589ad82d8514b"
server_key_iv: "d82b9fa5163089927bab6ed442c97fa4"
server_key_version: "encryptionkey@0"
I20260923 21:00:38.706523 3444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5"
format_stamp: "Formatted at 2026-09-23 21:00:38 on dist-test-slave-8xn5"
server_key: "55a278cc644d3afe59b589ad82d8514b"
server_key_iv: "d82b9fa5163089927bab6ed442c97fa4"
server_key_version: "encryptionkey@0"
I20260923 21:00:38.712563 3444 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:00:38.716674 3459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:38.718252 3444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 21:00:38.718525 3444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5"
format_stamp: "Formatted at 2026-09-23 21:00:38 on dist-test-slave-8xn5"
server_key: "55a278cc644d3afe59b589ad82d8514b"
server_key_iv: "d82b9fa5163089927bab6ed442c97fa4"
server_key_version: "encryptionkey@0"
I20260923 21:00:38.718752 3444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:38.730743 3444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:38.735342 3444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:38.735646 3444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:38.756835 3444 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41283
I20260923 21:00:38.756835 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41283 every 8 connection(s)
I20260923 21:00:38.758381 3444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:38.762542 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:38.762658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3444
I20260923 21:00:38.762850 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:38.763724 388 external_mini_cluster.cc:1428] Setting key 7f8852e64e6710d4739fa387a8f27b61
I20260923 21:00:38.778826 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5: Bootstrap starting.
Sep 23 21:00:38 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:38.782909 3511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:38.784168 3511 log.cc:824] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:38.787609 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5: No bootstrap required, opened a new log
I20260923 21:00:38.789464 3514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:38.765945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.97.62:41283)
0923 21:00:38.766899 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:38.766917 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:38.766962 (+ 45us) server_negotiation.cc:407] Connection header received
0923 21:00:38.769194 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:38.769221 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:38.769762 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:38.770153 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:38.772168 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:38.773482 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:38.775660 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:38.776437 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:38.776625 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:38.781776 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:38.781785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:38.781801 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:38.781848 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:38.785233 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:38.786007 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:38.786015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:38.786022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:38.786190 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:38.786634 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:38.786646 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:38.786649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:38.787194 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:38.787526 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:38.787954 (+ 428us) server_negotiation.cc:299] Negotiation successful
0923 21:00:38.788412 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":482,"threads_started":1}
I20260923 21:00:38.797092 3511 raft_consensus.cc:348] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41283 } }
I20260923 21:00:38.797559 3511 raft_consensus.cc:374] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:38.797648 3511 raft_consensus.cc:729] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c872a3fd7f1f4090a2e8044c0bf8d7f5, State: Initialized, Role: FOLLOWER
I20260923 21:00:38.798211 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41283 } }
I20260923 21:00:38.798480 3511 raft_consensus.cc:388] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:38.798686 3511 raft_consensus.cc:482] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:38.798923 3511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:38.801421 3511 raft_consensus.cc:504] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41283 } }
I20260923 21:00:38.802048 3511 leader_election.cc:302] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c872a3fd7f1f4090a2e8044c0bf8d7f5; no voters:
I20260923 21:00:38.802812 3511 leader_election.cc:288] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:38.802968 3516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:38.803742 3516 raft_consensus.cc:686] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [term 1 LEADER]: Becoming Leader. State: Replica: c872a3fd7f1f4090a2e8044c0bf8d7f5, State: Running, Role: LEADER
I20260923 21:00:38.804342 3516 consensus_queue.cc:237] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41283 } }
I20260923 21:00:38.805331 3511 sys_catalog.cc:564] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:38.807763 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c872a3fd7f1f4090a2e8044c0bf8d7f5. Latest consensus state: current_term: 1 leader_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41283 } } }
I20260923 21:00:38.808001 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c872a3fd7f1f4090a2e8044c0bf8d7f5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41283 } } }
I20260923 21:00:38.808250 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:38.808312 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:38.811129 3523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:38.816895 3523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:38.826586 3523 catalog_manager.cc:1347] Generated new cluster ID: d27be6cf27ea4dea8595c500af80e958
I20260923 21:00:38.826740 3523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:38.837766 3523 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:38.839365 3523 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:38.859629 3523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c872a3fd7f1f4090a2e8044c0bf8d7f5: Generated new TSK 0
I20260923 21:00:38.860702 3523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:38.927440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41283
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:39.290179 3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:39.290724 3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:39.290854 3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:39.299494 3537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:39.299626 3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:39.299741 3537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:39.310632 3537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:39.312261 3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:39.313620 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:39.328994 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:39.329162 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:39.329066 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:39.330256 3537 server_base.cc:1034] running on GCE node
I20260923 21:00:39.331013 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:39.332033 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:39.333243 3537 hybrid_clock.cc:648] HybridClock initialized: now 1790197239333189 us; error 54 us; skew 500 ppm
I20260923 21:00:39.333598 3537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:39 dist-test-slave-8xn5 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:39.337204 3537 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:39.339151 3537 webserver.cc:460] Webserver started at http://127.0.97.1:36587/ using document root <none> and password file <none>
I20260923 21:00:39.339943 3537 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:39.340095 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:39.340428 3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:39.343696 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "e591c07c728b4ffdb675eabaff31c5c4"
format_stamp: "Formatted at 2026-09-23 21:00:39 on dist-test-slave-8xn5"
server_key: "8e41d1a997578b5ae892190d969d61ed"
server_key_iv: "9c60fa6779c43bfa272077f7345e8d45"
server_key_version: "encryptionkey@0"
I20260923 21:00:39.344601 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e591c07c728b4ffdb675eabaff31c5c4"
format_stamp: "Formatted at 2026-09-23 21:00:39 on dist-test-slave-8xn5"
server_key: "8e41d1a997578b5ae892190d969d61ed"
server_key_iv: "9c60fa6779c43bfa272077f7345e8d45"
server_key_version: "encryptionkey@0"
I20260923 21:00:39.350531 3537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260923 21:00:39.354846 3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:39.356524 3537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:00:39.356740 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "e591c07c728b4ffdb675eabaff31c5c4"
format_stamp: "Formatted at 2026-09-23 21:00:39 on dist-test-slave-8xn5"
server_key: "8e41d1a997578b5ae892190d969d61ed"
server_key_iv: "9c60fa6779c43bfa272077f7345e8d45"
server_key_version: "encryptionkey@0"
I20260923 21:00:39.356962 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:39.371701 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:39.376828 3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:39.377185 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:39.378520 3537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:39.380090 3537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:39.380208 3537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:39.380393 3537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:39.380478 3537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:39.418202 3537 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46657
I20260923 21:00:39.418253 3664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46657 every 8 connection(s)
I20260923 21:00:39.419582 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:39.420071 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3537
I20260923 21:00:39.420281 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:39.421180 388 external_mini_cluster.cc:1428] Setting key a46bfb83bd7da170c2b83327bcb74bc7
Sep 23 21:00:39 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:00:39.442886 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:39.422793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49715 (local address 127.0.97.62:41283)
0923 21:00:39.425147 (+ 2354us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:39.425154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:39.425178 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:00:39.426706 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:39.426712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:39.426827 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:39.426957 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:39.428919 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.429674 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.431554 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.431984 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.432132 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:39.436181 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:39.436187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:39.436190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:39.436228 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:39.439325 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:39.440250 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:39.440254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:39.440257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:39.440389 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:39.440883 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:39.440887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:39.440889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:39.441199 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:39.441391 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:39.442176 (+ 785us) server_negotiation.cc:299] Negotiation successful
0923 21:00:39.442457 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2232,"thread_start_us":2117,"threads_started":1}
I20260923 21:00:39.444057 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:39.423530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41283 (local address 127.0.97.1:49715)
0923 21:00:39.424390 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:39.424459 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:39.426326 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:39.427304 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:39.427314 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:39.428035 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:39.428687 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:39.428707 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.429890 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.429894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:39.431367 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:39.431383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.432248 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.432255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:39.432524 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:39.433365 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:39.433395 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:39.435855 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:39.439545 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:39.439556 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:39.439573 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:39.440079 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:39.440516 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:39.440520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:39.440526 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:39.440741 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:39.441352 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:39.441362 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:39.441751 (+ 389us) client_negotiation.cc:769] Sending connection context
0923 21:00:39.442046 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 21:00:39.442748 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":343,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 21:00:39.446291 3665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41283
I20260923 21:00:39.446730 3665 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:39.447798 3665 heartbeater.cc:511] Master 127.0.97.62:41283 requested a full tablet report, sending...
I20260923 21:00:39.450759 3476 ts_manager.cc:194] Registered new tserver with Master: e591c07c728b4ffdb675eabaff31c5c4 (127.0.97.1:46657)
I20260923 21:00:39.452723 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:39.475049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41283
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:39.825017 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:39.825346 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:39.825444 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:39.832804 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:39.832934 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:39.833048 3671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:39.843729 3671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:39.845094 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:39.846462 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:39.861302 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:39.861357 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:39.861228 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:39.861963 3671 server_base.cc:1034] running on GCE node
I20260923 21:00:39.862716 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:39.863749 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:39.864995 3671 hybrid_clock.cc:648] HybridClock initialized: now 1790197239864923 us; error 65 us; skew 500 ppm
I20260923 21:00:39.865345 3671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:39 dist-test-slave-8xn5 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:39.868757 3671 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:39.870441 3671 webserver.cc:460] Webserver started at http://127.0.97.2:43241/ using document root <none> and password file <none>
I20260923 21:00:39.871200 3671 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:39.871349 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:39.871668 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:39.874922 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "9e554d377d88458f812d159daa6b2222"
format_stamp: "Formatted at 2026-09-23 21:00:39 on dist-test-slave-8xn5"
server_key: "f436d6289b04818c4bffbf9875d59ee8"
server_key_iv: "5713c784df3dc5adbdc5ed2e92eac1ab"
server_key_version: "encryptionkey@0"
I20260923 21:00:39.875752 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9e554d377d88458f812d159daa6b2222"
format_stamp: "Formatted at 2026-09-23 21:00:39 on dist-test-slave-8xn5"
server_key: "f436d6289b04818c4bffbf9875d59ee8"
server_key_iv: "5713c784df3dc5adbdc5ed2e92eac1ab"
server_key_version: "encryptionkey@0"
I20260923 21:00:39.881235 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:00:39.885077 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:39.886725 3671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:39.886915 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "9e554d377d88458f812d159daa6b2222"
format_stamp: "Formatted at 2026-09-23 21:00:39 on dist-test-slave-8xn5"
server_key: "f436d6289b04818c4bffbf9875d59ee8"
server_key_iv: "5713c784df3dc5adbdc5ed2e92eac1ab"
server_key_version: "encryptionkey@0"
I20260923 21:00:39.887137 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:39.909737 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:39.914572 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:39.914939 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:39.915966 3671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:39.917425 3671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:39.917539 3671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:39.917683 3671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:39.917754 3671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:39.952922 3671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45873
I20260923 21:00:39.952997 3798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45873 every 8 connection(s)
I20260923 21:00:39.954280 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:39.956023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3671
I20260923 21:00:39.956197 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:39.956636 388 external_mini_cluster.cc:1428] Setting key de1cfc02b12eaba661d595b25fffb4c2
Sep 23 21:00:39 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:00:39.979242 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:39.957360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35055 (local address 127.0.97.62:41283)
0923 21:00:39.957851 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:39.957857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:39.959135 (+ 1278us) server_negotiation.cc:407] Connection header received
0923 21:00:39.961051 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:39.961056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:39.961152 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:39.961281 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:39.963139 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.964249 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.966087 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.966877 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.967036 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:39.971289 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:39.971298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:39.971305 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:39.971359 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:39.975356 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:39.976340 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:39.976345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:39.976348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:39.976485 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:39.977083 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:39.977088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:39.977090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:39.977394 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:39.977530 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:39.978613 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0923 21:00:39.978945 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260923 21:00:39.980695 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:39.958028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41283 (local address 127.0.97.2:35055)
0923 21:00:39.958971 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:39.959026 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:39.960717 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:39.961591 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:39.961601 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:39.962253 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:39.962955 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:39.962980 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.964436 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.964440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:39.965873 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:39.965887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:39.967108 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:39.967113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:39.967344 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:39.968208 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:39.968353 (+ 145us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:39.970958 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:39.975580 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:39.975589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:39.975607 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:39.976186 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:39.976652 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:39.976661 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:39.976668 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:39.976924 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:39.977542 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:39.977550 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:39.978063 (+ 513us) client_negotiation.cc:769] Sending connection context
0923 21:00:39.978500 (+ 437us) client_negotiation.cc:240] Negotiation successful
0923 21:00:39.979054 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":407,"threads_started":1}
I20260923 21:00:39.982676 3799 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41283
I20260923 21:00:39.983024 3799 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:39.983845 3799 heartbeater.cc:511] Master 127.0.97.62:41283 requested a full tablet report, sending...
I20260923 21:00:39.985757 3476 ts_manager.cc:194] Registered new tserver with Master: 9e554d377d88458f812d159daa6b2222 (127.0.97.2:45873)
I20260923 21:00:39.986899 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:40.006731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41283
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:40.358140 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:40.358609 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:40.358773 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:40.366551 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:40.366704 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:40.366883 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:40.377204 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:40.378670 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:40.380038 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:40.394645 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:40.394683 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:40.394606 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:40.395880 3805 server_base.cc:1034] running on GCE node
I20260923 21:00:40.396507 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:40.397519 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:40.398814 3805 hybrid_clock.cc:648] HybridClock initialized: now 1790197240398726 us; error 81 us; skew 500 ppm
I20260923 21:00:40.399217 3805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:40.403210 3805 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:40.404949 3805 webserver.cc:460] Webserver started at http://127.0.97.3:46199/ using document root <none> and password file <none>
I20260923 21:00:40.405879 3805 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:40.406000 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:40.406409 3805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:40.409914 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "461ca151a801471d879c99f4bb8ea3c5"
format_stamp: "Formatted at 2026-09-23 21:00:40 on dist-test-slave-8xn5"
server_key: "085f9feaf38799738525901c2b4f4683"
server_key_iv: "3045e9ce9f973dc434e401bb5db95f20"
server_key_version: "encryptionkey@0"
I20260923 21:00:40.410981 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "461ca151a801471d879c99f4bb8ea3c5"
format_stamp: "Formatted at 2026-09-23 21:00:40 on dist-test-slave-8xn5"
server_key: "085f9feaf38799738525901c2b4f4683"
server_key_iv: "3045e9ce9f973dc434e401bb5db95f20"
server_key_version: "encryptionkey@0"
I20260923 21:00:40.417517 3805 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:00:40.422775 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:40.424582 3805 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:40.424734 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "461ca151a801471d879c99f4bb8ea3c5"
format_stamp: "Formatted at 2026-09-23 21:00:40 on dist-test-slave-8xn5"
server_key: "085f9feaf38799738525901c2b4f4683"
server_key_iv: "3045e9ce9f973dc434e401bb5db95f20"
server_key_version: "encryptionkey@0"
I20260923 21:00:40.424918 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:40.452551 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:40.456563 3665 heartbeater.cc:503] Master 127.0.97.62:41283 was elected leader, sending a full tablet report...
I20260923 21:00:40.457744 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:40.458087 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:40.459223 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:40.460599 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:40.460675 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:40.460779 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:40.460824 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:40.496094 3805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36935
I20260923 21:00:40.496207 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36935 every 8 connection(s)
I20260923 21:00:40.497608 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:40.507938 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3805
I20260923 21:00:40.508420 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:40.509159 388 external_mini_cluster.cc:1428] Setting key 2275b5c0d9adb359af0fba3601656ca9
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:40.531149 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:40.500660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52397 (local address 127.0.97.62:41283)
0923 21:00:40.501244 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:40.501249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:40.502203 (+ 954us) server_negotiation.cc:407] Connection header received
0923 21:00:40.506236 (+ 4033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:40.506244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:40.506430 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:40.506782 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:40.509926 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.511454 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.515257 (+ 3803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.516077 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.516276 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:40.521816 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:40.521822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:40.521826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:40.521881 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:40.526037 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.527567 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.527574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.527577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.527746 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.528456 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.528460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.528463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.528822 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:40.528996 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:40.530594 (+ 1598us) server_negotiation.cc:299] Negotiation successful
0923 21:00:40.530861 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":322,"threads_started":1}
I20260923 21:00:40.532186 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:40.501162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41283 (local address 127.0.97.3:52397)
0923 21:00:40.502003 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:40.502056 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:40.505689 (+ 3633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:40.507088 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:40.507100 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:40.507945 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:40.509278 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:40.509381 (+ 103us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.511987 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.511997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:40.515012 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:40.515047 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.516356 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.516362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:40.516671 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:40.517787 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:40.517816 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:40.521422 (+ 3606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:40.526267 (+ 4845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:40.526283 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:40.526312 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:40.527207 (+ 895us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:40.527912 (+ 705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:40.527917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:40.527921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:40.528266 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:40.529003 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:40.529015 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:40.529583 (+ 568us) client_negotiation.cc:769] Sending connection context
0923 21:00:40.529983 (+ 400us) client_negotiation.cc:240] Negotiation successful
0923 21:00:40.530690 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
I20260923 21:00:40.534176 3933 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41283
I20260923 21:00:40.534593 3933 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:40.535570 3933 heartbeater.cc:511] Master 127.0.97.62:41283 requested a full tablet report, sending...
I20260923 21:00:40.537549 3476 ts_manager.cc:194] Registered new tserver with Master: 461ca151a801471d879c99f4bb8ea3c5 (127.0.97.3:36935)
I20260923 21:00:40.538460 3476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52397
I20260923 21:00:40.541114 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:40.602686 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:40.585544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.97.62:41283)
0923 21:00:40.585729 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:40.585734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:40.586407 (+ 673us) server_negotiation.cc:407] Connection header received
0923 21:00:40.586650 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:40.586654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:40.586724 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:40.586824 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:40.588704 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.590054 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.591540 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.592088 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.592210 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:40.596458 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:40.596465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:40.596469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:40.596516 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:40.599498 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.600244 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.600249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.600251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.600464 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.600826 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.600829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.600831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.601191 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:40.601463 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:40.601997 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 21:00:40.602200 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 21:00:40.603494 3476 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53450
I20260923 21:00:40.605628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3537
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:40.623965 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:40.607113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48014 (local address 127.0.97.1:46657)
0923 21:00:40.607863 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:40.607872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:40.607901 (+ 29us) server_negotiation.cc:407] Connection header received
0923 21:00:40.607966 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:40.607973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:40.608201 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:40.608349 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:40.609645 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.611104 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.612423 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.613080 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.613205 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:40.617732 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:40.617747 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:40.617763 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:40.617812 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:40.620860 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.621654 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.621662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.621664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.621795 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.622238 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.622241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.622243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.622719 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:40.622947 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:40.623384 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 21:00:40.623670 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":393,"threads_started":1}
W20260923 21:00:40.625366 3660 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48014
W20260923 21:00:40.626698 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:40.626904 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3537
I20260923 21:00:40.642553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3671
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:00:40.660579 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:40.643548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.0.97.2:45873)
0923 21:00:40.644153 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:40.644161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:40.644188 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:00:40.644250 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:40.644257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:40.644478 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:40.644656 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:40.646022 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.647217 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.648603 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.649198 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.649357 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:40.653756 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:40.653773 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:40.653791 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:40.653844 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:40.657167 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.658110 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.658120 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.658123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.658281 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.658819 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.658826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.658830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.659220 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:40.659593 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:40.659892 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 21:00:40.660206 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":276,"threads_started":1}
W20260923 21:00:40.662421 3794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34626
W20260923 21:00:40.663851 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3671: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:40.664009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3671
I20260923 21:00:40.679566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3805
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197240, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:00:40.699579 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:40.680591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.0.97.3:36935)
0923 21:00:40.681270 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:40.681278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:40.681310 (+ 32us) server_negotiation.cc:407] Connection header received
0923 21:00:40.681383 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:40.681390 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:40.681634 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:40.681835 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:40.683524 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.684747 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.686301 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:40.687154 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:40.687359 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:40.692718 (+ 5359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:40.692742 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:40.692761 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:40.692808 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:40.696576 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.697411 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.697419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.697421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.697553 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:40.697905 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:40.697908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:40.697910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:40.698265 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:40.698615 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:40.698888 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 21:00:40.699110 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":363,"threads_started":1}
W20260923 21:00:40.701007 3928 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47688
W20260923 21:00:40.702277 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:00:40.702492 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3805
I20260923 21:00:40.718426 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3444
I20260923 21:00:40.719731 3506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:40.824015 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3444
2026-09-23T21:00:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3843 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3951](info): set up 2 sockets
Sep 23 21:00:40 dist-test-slave-8xn5 krb5kdc[3951](info): commencing operation
krb5kdc: starting...
W20260923 21:00:41.970942 388 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:42 dist-test-slave-8xn5 krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:42.185791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34637
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:42.539906 3964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:42.540280 3964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:42.540385 3964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:42.548853 3964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:42.548987 3964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:42.549036 3964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:42.549134 3964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:42.549185 3964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:42.559684 3964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:42.561280 3964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:42.562772 3964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:42.576072 3969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:42.576052 3972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:42.576006 3970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:42.577020 3964 server_base.cc:1034] running on GCE node
I20260923 21:00:42.577872 3964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:42.579559 3964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:42.580780 3964 hybrid_clock.cc:648] HybridClock initialized: now 1790197242580725 us; error 49 us; skew 500 ppm
I20260923 21:00:42.581117 3964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:42 dist-test-slave-8xn5 krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:42.589823 3964 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:42.591657 3964 webserver.cc:460] Webserver started at http://127.0.97.62:40015/ using document root <none> and password file <none>
I20260923 21:00:42.592525 3964 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:42.592691 3964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:42.593039 3964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:42.596478 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "46472e029ff6420d8d66f32786d6dc50"
format_stamp: "Formatted at 2026-09-23 21:00:42 on dist-test-slave-8xn5"
server_key: "ab7911d5b22767723c9d9b0c4317ed35"
server_key_iv: "183179f44654929aeb3c8ac0e0b4a849"
server_key_version: "encryptionkey@0"
I20260923 21:00:42.597350 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "46472e029ff6420d8d66f32786d6dc50"
format_stamp: "Formatted at 2026-09-23 21:00:42 on dist-test-slave-8xn5"
server_key: "ab7911d5b22767723c9d9b0c4317ed35"
server_key_iv: "183179f44654929aeb3c8ac0e0b4a849"
server_key_version: "encryptionkey@0"
I20260923 21:00:42.603451 3964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 21:00:42.607589 3979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:42.609314 3964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:00:42.609586 3964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "46472e029ff6420d8d66f32786d6dc50"
format_stamp: "Formatted at 2026-09-23 21:00:42 on dist-test-slave-8xn5"
server_key: "ab7911d5b22767723c9d9b0c4317ed35"
server_key_iv: "183179f44654929aeb3c8ac0e0b4a849"
server_key_version: "encryptionkey@0"
I20260923 21:00:42.609817 3964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:42.624308 3964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:42.629288 3964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:42.629612 3964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:42.649152 3964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34637
I20260923 21:00:42.649147 4030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34637 every 8 connection(s)
I20260923 21:00:42.650715 3964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:42.654927 4031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:42.657750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 3964
I20260923 21:00:42.657974 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:42.658809 388 external_mini_cluster.cc:1428] Setting key 81533bff980d4d5816b7b126693dc71f
I20260923 21:00:42.669059 4031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50: Bootstrap starting.
I20260923 21:00:42.673360 4031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:42.675035 4031 log.cc:824] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:00:42 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:42.679015 4031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50: No bootstrap required, opened a new log
I20260923 21:00:42.685704 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:42.661240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.0.97.62:34637)
0923 21:00:42.662269 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:42.662284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:42.662397 (+ 113us) server_negotiation.cc:407] Connection header received
0923 21:00:42.664352 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:42.664383 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:42.664921 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:42.665328 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:42.667330 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:42.668743 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:42.670577 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:42.671325 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:42.671500 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:42.676856 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:42.676870 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:42.676896 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:42.676940 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:42.680987 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:42.681972 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:42.681979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:42.681987 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:42.682177 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:42.682664 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:42.682668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:42.682670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:42.683383 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:42.683622 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:42.684098 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 21:00:42.684484 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":438,"threads_started":1}
I20260923 21:00:42.689110 4031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46472e029ff6420d8d66f32786d6dc50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260923 21:00:42.689580 4031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:42.689692 4031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46472e029ff6420d8d66f32786d6dc50, State: Initialized, Role: FOLLOWER
I20260923 21:00:42.690203 4031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46472e029ff6420d8d66f32786d6dc50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260923 21:00:42.690477 4031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:42.690579 4031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:42.690707 4031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:42.693012 4031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46472e029ff6420d8d66f32786d6dc50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260923 21:00:42.693661 4031 leader_election.cc:302] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46472e029ff6420d8d66f32786d6dc50; no voters:
I20260923 21:00:42.694425 4031 leader_election.cc:288] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:42.694545 4036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:42.695293 4036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [term 1 LEADER]: Becoming Leader. State: Replica: 46472e029ff6420d8d66f32786d6dc50, State: Running, Role: LEADER
I20260923 21:00:42.695847 4036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46472e029ff6420d8d66f32786d6dc50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } }
I20260923 21:00:42.696746 4031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:42.698791 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46472e029ff6420d8d66f32786d6dc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46472e029ff6420d8d66f32786d6dc50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } } }
I20260923 21:00:42.699007 4038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46472e029ff6420d8d66f32786d6dc50. Latest consensus state: current_term: 1 leader_uuid: "46472e029ff6420d8d66f32786d6dc50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46472e029ff6420d8d66f32786d6dc50" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34637 } } }
I20260923 21:00:42.699321 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:42.699321 4038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:42.700481 4042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:42.709869 4042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:42.721097 4042 catalog_manager.cc:1347] Generated new cluster ID: 5b97edffef6e499eb87b7cb6b38583c8
I20260923 21:00:42.721374 4042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:42.738511 4042 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:42.739991 4042 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:42.747503 4042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46472e029ff6420d8d66f32786d6dc50: Generated new TSK 0
I20260923 21:00:42.748473 4042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:42.811779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:43.187801 4057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:43.188184 4057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:43.188359 4057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:43.196208 4057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:43.196409 4057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:43.196599 4057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:43.207988 4057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:43.209457 4057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:43.211092 4057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:43.226357 4062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:43.226418 4065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:43.226441 4063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:43.227371 4057 server_base.cc:1034] running on GCE node
I20260923 21:00:43.228024 4057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:43.229053 4057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:43.230376 4057 hybrid_clock.cc:648] HybridClock initialized: now 1790197243230275 us; error 47 us; skew 500 ppm
I20260923 21:00:43.230748 4057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:43 dist-test-slave-8xn5 krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:43.234262 4057 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:43.235982 4057 webserver.cc:460] Webserver started at http://127.0.97.1:45805/ using document root <none> and password file <none>
I20260923 21:00:43.236739 4057 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:43.236896 4057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:43.237218 4057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:43.240391 4057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "dc684c9e25bb45a9bf2608b18f11faad"
format_stamp: "Formatted at 2026-09-23 21:00:43 on dist-test-slave-8xn5"
server_key: "12ab5cec20ac343f6e64d4e3961f6e2b"
server_key_iv: "6c549524666fa0f2fcc26cd912c2e73a"
server_key_version: "encryptionkey@0"
I20260923 21:00:43.241336 4057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dc684c9e25bb45a9bf2608b18f11faad"
format_stamp: "Formatted at 2026-09-23 21:00:43 on dist-test-slave-8xn5"
server_key: "12ab5cec20ac343f6e64d4e3961f6e2b"
server_key_iv: "6c549524666fa0f2fcc26cd912c2e73a"
server_key_version: "encryptionkey@0"
I20260923 21:00:43.247237 4057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 21:00:43.251330 4072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:43.252935 4057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 21:00:43.253156 4057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "dc684c9e25bb45a9bf2608b18f11faad"
format_stamp: "Formatted at 2026-09-23 21:00:43 on dist-test-slave-8xn5"
server_key: "12ab5cec20ac343f6e64d4e3961f6e2b"
server_key_iv: "6c549524666fa0f2fcc26cd912c2e73a"
server_key_version: "encryptionkey@0"
I20260923 21:00:43.253404 4057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:43.274307 4057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:43.279316 4057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:43.279651 4057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:43.280750 4057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:43.282238 4057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:43.282373 4057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:43.282523 4057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:43.282593 4057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:43.317394 4057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40437
I20260923 21:00:43.317441 4184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40437 every 8 connection(s)
I20260923 21:00:43.318758 4057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:43.327553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 4057
I20260923 21:00:43.327805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:43.328564 388 external_mini_cluster.cc:1428] Setting key 388176c60a861e15444efec9bc354401
Sep 23 21:00:43 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:43.342730 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:43.321574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49167 (local address 127.0.97.62:34637)
0923 21:00:43.322094 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:43.322098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:43.323002 (+ 904us) server_negotiation.cc:407] Connection header received
0923 21:00:43.324999 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:43.325004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:43.325098 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:43.325218 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:43.326950 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.328121 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.330685 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.331289 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.331428 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:43.335197 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:43.335205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:43.335213 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:43.335260 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:43.338955 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:43.339815 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:43.339825 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:43.339831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:43.339980 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:43.340484 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:43.340492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:43.340499 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:43.340893 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:43.341046 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:43.341933 (+ 887us) server_negotiation.cc:299] Negotiation successful
0923 21:00:43.342210 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":308,"threads_started":1}
I20260923 21:00:43.343672 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:43.321996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34637 (local address 127.0.97.1:49167)
0923 21:00:43.322830 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:43.322877 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:43.324697 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:43.325483 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:43.325492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:43.326061 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:43.326761 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:43.326780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.328314 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.328318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:43.330536 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:43.330555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.331555 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.331560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:43.331765 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:43.332540 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:43.332567 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:43.334894 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:43.339137 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:43.339145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:43.339162 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:43.339660 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:43.340124 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:43.340128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:43.340140 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:43.340355 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:43.341069 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:43.341077 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:43.341481 (+ 404us) client_negotiation.cc:769] Sending connection context
0923 21:00:43.341786 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 21:00:43.342256 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":362,"threads_started":1}
I20260923 21:00:43.345572 4185 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34637
I20260923 21:00:43.345958 4185 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:43.347038 4185 heartbeater.cc:511] Master 127.0.97.62:34637 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:00:43.349545 3996 ts_manager.cc:194] Registered new tserver with Master: dc684c9e25bb45a9bf2608b18f11faad (127.0.97.1:40437)
I20260923 21:00:43.351263 3996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:43.378721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:43.737116 4191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:43.737516 4191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:43.737720 4191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:43.745437 4191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:43.745645 4191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:43.745839 4191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:43.757032 4191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:43.758628 4191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:43.760089 4191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:43.775477 4196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:43.775539 4197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:43.776088 4199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:43.776774 4191 server_base.cc:1034] running on GCE node
I20260923 21:00:43.777387 4191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:43.778537 4191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:43.779747 4191 hybrid_clock.cc:648] HybridClock initialized: now 1790197243779681 us; error 57 us; skew 500 ppm
I20260923 21:00:43.780122 4191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:43 dist-test-slave-8xn5 krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:43.783558 4191 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:43.785214 4191 webserver.cc:460] Webserver started at http://127.0.97.2:42437/ using document root <none> and password file <none>
I20260923 21:00:43.785986 4191 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:43.786141 4191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:43.786535 4191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:43.789666 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "b8e9bb74a9b54b97aa1acc8178792910"
format_stamp: "Formatted at 2026-09-23 21:00:43 on dist-test-slave-8xn5"
server_key: "e1a5194221afc3ce8e3e333024c09aff"
server_key_iv: "ca6342c66aeb23845987ae26aba96c25"
server_key_version: "encryptionkey@0"
I20260923 21:00:43.790627 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b8e9bb74a9b54b97aa1acc8178792910"
format_stamp: "Formatted at 2026-09-23 21:00:43 on dist-test-slave-8xn5"
server_key: "e1a5194221afc3ce8e3e333024c09aff"
server_key_iv: "ca6342c66aeb23845987ae26aba96c25"
server_key_version: "encryptionkey@0"
I20260923 21:00:43.796607 4191 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:00:43.800644 4206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:43.802253 4191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:00:43.802505 4191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "b8e9bb74a9b54b97aa1acc8178792910"
format_stamp: "Formatted at 2026-09-23 21:00:43 on dist-test-slave-8xn5"
server_key: "e1a5194221afc3ce8e3e333024c09aff"
server_key_iv: "ca6342c66aeb23845987ae26aba96c25"
server_key_version: "encryptionkey@0"
I20260923 21:00:43.802776 4191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:43.831914 4191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:43.836524 4191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:43.836828 4191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:43.837764 4191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:43.839299 4191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:43.839377 4191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:43.839464 4191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:43.839507 4191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:43.874459 4191 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41733
I20260923 21:00:43.874643 4318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41733 every 8 connection(s)
I20260923 21:00:43.875918 4191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:43.880321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 4191
I20260923 21:00:43.880544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:43.881160 388 external_mini_cluster.cc:1428] Setting key cb8f33680b85e9e4a414191a0eeab0d5
Sep 23 21:00:43 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:43.901098 4321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:43.878645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33289 (local address 127.0.97.62:34637)
0923 21:00:43.879122 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:43.879127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:43.880152 (+ 1025us) server_negotiation.cc:407] Connection header received
0923 21:00:43.882635 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:43.882641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:43.882712 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:43.882841 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:43.885275 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.886069 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.888724 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.889260 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.889403 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:43.893027 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:43.893036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:43.893044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:43.893093 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:43.896968 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:43.898212 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:43.898221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:43.898237 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:43.898464 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:43.898975 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:43.898983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:43.898988 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:43.899417 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:43.899601 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:43.900381 (+ 780us) server_negotiation.cc:299] Negotiation successful
0923 21:00:43.900651 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:00:43.901959 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:43.879132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34637 (local address 127.0.97.2:33289)
0923 21:00:43.879942 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:43.879996 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:43.882273 (+ 2277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:43.883281 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:43.883295 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:43.884123 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:43.885053 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:43.885086 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.886248 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.886256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:43.888565 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:43.888590 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:43.889500 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:43.889506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:43.889748 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:43.890492 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:43.890517 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:43.892751 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:43.897211 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:43.897219 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:43.897235 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:43.898003 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:43.898587 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:43.898591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:43.898594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:43.898813 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:43.899613 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:43.899619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:43.899981 (+ 362us) client_negotiation.cc:769] Sending connection context
0923 21:00:43.900217 (+ 236us) client_negotiation.cc:240] Negotiation successful
0923 21:00:43.900697 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":315,"threads_started":1}
I20260923 21:00:43.903705 4319 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34637
I20260923 21:00:43.904052 4319 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:43.904937 4319 heartbeater.cc:511] Master 127.0.97.62:34637 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:00:43.906697 3996 ts_manager.cc:194] Registered new tserver with Master: b8e9bb74a9b54b97aa1acc8178792910 (127.0.97.2:41733)
I20260923 21:00:43.907560 3996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:43.931098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:44.286545 4325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:44.286918 4325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:44.287084 4325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:44.294682 4325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:44.294811 4325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:44.295017 4325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:44.305542 4325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:44.307030 4325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:44.308442 4325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:44.322500 4331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:44.322728 4330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:44.323036 4333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:44.323302 4325 server_base.cc:1034] running on GCE node
I20260923 21:00:44.323964 4325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:44.324947 4325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:44.326267 4325 hybrid_clock.cc:648] HybridClock initialized: now 1790197244326205 us; error 49 us; skew 500 ppm
I20260923 21:00:44.326689 4325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:44 dist-test-slave-8xn5 krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:44.330314 4325 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:44.332062 4325 webserver.cc:460] Webserver started at http://127.0.97.3:44081/ using document root <none> and password file <none>
I20260923 21:00:44.332798 4325 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:44.332928 4325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:44.333257 4325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:44.336416 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "7a135da234eb45b9a39c1a4bbdbdae03"
format_stamp: "Formatted at 2026-09-23 21:00:44 on dist-test-slave-8xn5"
server_key: "f622cf35a9ce515814b7f2bfa3988cc1"
server_key_iv: "53c6e66fda534ad26bb0cb427ced1c08"
server_key_version: "encryptionkey@0"
I20260923 21:00:44.337273 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7a135da234eb45b9a39c1a4bbdbdae03"
format_stamp: "Formatted at 2026-09-23 21:00:44 on dist-test-slave-8xn5"
server_key: "f622cf35a9ce515814b7f2bfa3988cc1"
server_key_iv: "53c6e66fda534ad26bb0cb427ced1c08"
server_key_version: "encryptionkey@0"
I20260923 21:00:44.343693 4325 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 21:00:44.347968 4340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:44.349713 4325 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:00:44.349933 4325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "7a135da234eb45b9a39c1a4bbdbdae03"
format_stamp: "Formatted at 2026-09-23 21:00:44 on dist-test-slave-8xn5"
server_key: "f622cf35a9ce515814b7f2bfa3988cc1"
server_key_iv: "53c6e66fda534ad26bb0cb427ced1c08"
server_key_version: "encryptionkey@0"
I20260923 21:00:44.350169 4325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:44.355404 4185 heartbeater.cc:503] Master 127.0.97.62:34637 was elected leader, sending a full tablet report...
I20260923 21:00:44.371899 4325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:44.376756 4325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:44.377089 4325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:44.378216 4325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:44.379722 4325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:44.379832 4325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:44.379987 4325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:44.380056 4325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:44.415463 4325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39867
I20260923 21:00:44.415511 4452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39867 every 8 connection(s)
I20260923 21:00:44.416891 4325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:44.422881 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 4325
I20260923 21:00:44.423075 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:44.423595 388 external_mini_cluster.cc:1428] Setting key dc08e51f83e47b723e9dd89589b2a6eb
Sep 23 21:00:44 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:44.438835 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:44.419595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42495 (local address 127.0.97.62:34637)
0923 21:00:44.420051 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:44.420056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:44.421079 (+ 1023us) server_negotiation.cc:407] Connection header received
0923 21:00:44.423190 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:44.423197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:44.423314 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:44.423445 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:44.425052 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.426064 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.427811 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.428256 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.428402 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:44.432145 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:44.432151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:44.432154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:44.432191 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:44.435226 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.436137 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.436142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.436145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.436279 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.436764 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.436775 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.436778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.437062 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:44.437237 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:44.438081 (+ 844us) server_negotiation.cc:299] Negotiation successful
0923 21:00:44.438379 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260923 21:00:44.440001 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:44.420115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34637 (local address 127.0.97.3:42495)
0923 21:00:44.420886 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:44.420930 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:44.422786 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:44.423694 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:44.423712 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:44.424269 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:44.424898 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:44.424917 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.426215 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.426221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:44.427611 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:44.427633 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.428511 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.428518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:44.428771 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:44.429461 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:44.429487 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:44.431883 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:44.435422 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:44.435434 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:44.435458 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:44.435966 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:44.436407 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:44.436412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:44.436416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:44.436627 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:44.437202 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:44.437212 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:44.437660 (+ 448us) client_negotiation.cc:769] Sending connection context
0923 21:00:44.437976 (+ 316us) client_negotiation.cc:240] Negotiation successful
0923 21:00:44.438660 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":298,"threads_started":1}
I20260923 21:00:44.441431 4453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34637
I20260923 21:00:44.441815 4453 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:44.442799 4453 heartbeater.cc:511] Master 127.0.97.62:34637 requested a full tablet report, sending...
I20260923 21:00:44.444345 3996 ts_manager.cc:194] Registered new tserver with Master: 7a135da234eb45b9a39c1a4bbdbdae03 (127.0.97.3:39867)
I20260923 21:00:44.445169 3996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42495
I20260923 21:00:44.451891 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 21:00:44 dist-test-slave-8xn5 krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 21:00:44 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:44.505352 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:44.487325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.0.97.1:40437)
0923 21:00:44.487983 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:44.487990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:44.488019 (+ 29us) server_negotiation.cc:407] Connection header received
0923 21:00:44.488081 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:44.488088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:44.488312 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:44.488455 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:44.490054 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.491292 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.492882 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.493507 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.493625 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:44.498619 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:44.498635 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:44.498653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:44.498697 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:44.501875 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.502797 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.502806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.502808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.503004 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.503442 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.503448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.503451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.503893 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:44.504230 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:44.504646 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 21:00:44.504954 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":321,"threads_started":1}
Sep 23 21:00:44 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:44.526940 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:44.510875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.0.97.62:34637)
0923 21:00:44.511088 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:44.511094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:44.511249 (+ 155us) server_negotiation.cc:407] Connection header received
0923 21:00:44.511411 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:44.511415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:44.511480 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:44.511567 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:44.513104 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.513951 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.515709 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.516165 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.516276 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:44.520951 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:44.520957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:44.520960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:44.521004 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:44.523887 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.524642 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.524646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.524648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.524820 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.525202 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.525206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.525208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.525650 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:44.525865 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:44.526465 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 21:00:44.526675 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 21:00:44.527710 3996 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35878
I20260923 21:00:44.528920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 4057
I20260923 21:00:44.543453 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:44.529982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.97.1:40437)
0923 21:00:44.530167 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:44.530171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:44.530526 (+ 355us) server_negotiation.cc:407] Connection header received
0923 21:00:44.530679 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:44.530683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:44.530777 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:44.530881 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:44.532385 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.533084 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.534613 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.535060 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.535162 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:44.537716 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:44.537721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:44.537725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:44.537761 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:44.540582 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.541348 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.541353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.541355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.541512 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.541914 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.541918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.541921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.542231 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:44.542548 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:44.542971 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 21:00:44.543190 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:00:44.544201 4180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:44.634505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 4057
I20260923 21:00:44.651815 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 4191
Sep 23 21:00:44 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:00:44.669597 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:44.652745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.0.97.2:41733)
0923 21:00:44.653373 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:44.653380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:44.653407 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:00:44.653468 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:44.653475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:44.653704 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:44.653877 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:44.655532 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.656777 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.658136 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.658932 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.659075 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:44.663465 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:44.663480 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:44.663497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:44.663540 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:44.666803 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.667619 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.667626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.667629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.667754 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.668083 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.668086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.668088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.668502 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:44.668742 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:44.668976 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 21:00:44.669182 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":327,"threads_started":1}
I20260923 21:00:44.671240 4314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:44.756763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 4191
I20260923 21:00:44.773432 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 4325
Sep 23 21:00:44 dist-test-slave-8xn5 krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:00:44.794564 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:44.774858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.97.3:39867)
0923 21:00:44.775624 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:44.775631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:44.775664 (+ 33us) server_negotiation.cc:407] Connection header received
0923 21:00:44.775730 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:44.775736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:44.776007 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:44.776209 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:44.777594 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.778735 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.780120 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:44.780741 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:44.780906 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:44.786976 (+ 6070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:44.786996 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:44.787014 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:44.787067 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:44.790945 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.791943 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.791952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.791955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.792135 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:44.792583 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:44.792590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:44.792593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:44.793036 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:44.793402 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:44.793739 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 21:00:44.794044 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":444,"threads_started":1}
I20260923 21:00:44.795960 4448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:44.881750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 4325
I20260923 21:00:44.898267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 3964
I20260923 21:00:44.899626 4026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:45.004925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 3964
2026-09-23T21:00:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4181 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:45 dist-test-slave-8xn5 krb5kdc[4473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:45 dist-test-slave-8xn5 krb5kdc[4473](info): set up 2 sockets
Sep 23 21:00:45 dist-test-slave-8xn5 krb5kdc[4473](info): commencing operation
krb5kdc: starting...
W20260923 21:00:46.187052 388 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:46 dist-test-slave-8xn5 krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:46.412384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36071
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:46.762887 4486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:46.763213 4486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:46.763309 4486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:46.771031 4486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:46.771169 4486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:46.771219 4486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:46.771277 4486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:46.771325 4486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:46.781635 4486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:46.783165 4486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:46.784621 4486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:46.797097 4491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:46.797051 4492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:46.797128 4494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:46.798051 4486 server_base.cc:1034] running on GCE node
I20260923 21:00:46.798971 4486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:46.800535 4486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:46.801832 4486 hybrid_clock.cc:648] HybridClock initialized: now 1790197246801730 us; error 50 us; skew 500 ppm
I20260923 21:00:46.802210 4486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:46 dist-test-slave-8xn5 krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:46.805622 4486 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:46.806890 4486 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:46.806985 4486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:46.807262 4486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:46.810482 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "2fde392861e6404e8141a608f37de09c"
format_stamp: "Formatted at 2026-09-23 21:00:46 on dist-test-slave-8xn5"
server_key: "0eab467f196dabcacf18dc3298def7aa"
server_key_iv: "b0d0fdaea4464d1014e010d0925dc24f"
server_key_version: "encryptionkey@0"
I20260923 21:00:46.811265 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "2fde392861e6404e8141a608f37de09c"
format_stamp: "Formatted at 2026-09-23 21:00:46 on dist-test-slave-8xn5"
server_key: "0eab467f196dabcacf18dc3298def7aa"
server_key_iv: "b0d0fdaea4464d1014e010d0925dc24f"
server_key_version: "encryptionkey@0"
I20260923 21:00:46.816538 4486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:00:46.820463 4500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:46.822059 4486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:00:46.822288 4486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "2fde392861e6404e8141a608f37de09c"
format_stamp: "Formatted at 2026-09-23 21:00:46 on dist-test-slave-8xn5"
server_key: "0eab467f196dabcacf18dc3298def7aa"
server_key_iv: "b0d0fdaea4464d1014e010d0925dc24f"
server_key_version: "encryptionkey@0"
I20260923 21:00:46.822603 4486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:46.872570 4486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:46.877609 4486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:46.877959 4486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:46.899125 4486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36071
I20260923 21:00:46.899155 4551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36071 every 8 connection(s)
I20260923 21:00:46.900164 4486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:46.904493 4552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:46.904691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 4486
I20260923 21:00:46.904969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:46.905697 388 external_mini_cluster.cc:1428] Setting key 24816c55334781e0e532f618b2f4dd80
I20260923 21:00:46.919548 4552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c: Bootstrap starting.
Sep 23 21:00:46 dist-test-slave-8xn5 krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:46.922945 4552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:46.924197 4552 log.cc:824] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:46.928560 4552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c: No bootstrap required, opened a new log
I20260923 21:00:46.933215 4555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:46.908290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53132 (local address 127.0.97.62:36071)
0923 21:00:46.909161 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:46.909187 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:46.909241 (+ 54us) server_negotiation.cc:407] Connection header received
0923 21:00:46.911468 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:46.911501 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:46.912116 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:46.912623 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:46.914249 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:46.916391 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:46.917985 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:46.919098 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:46.919306 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:46.924029 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:46.924041 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:46.924063 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:46.924117 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:46.928147 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:46.929043 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:46.929064 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:46.929076 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:46.929302 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:46.929691 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:46.929696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:46.929699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:46.930466 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:46.930832 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:46.931157 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 21:00:46.931675 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":384,"threads_started":1}
I20260923 21:00:46.938668 4552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fde392861e6404e8141a608f37de09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36071 } }
I20260923 21:00:46.939267 4552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:46.939373 4552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fde392861e6404e8141a608f37de09c, State: Initialized, Role: FOLLOWER
I20260923 21:00:46.940042 4552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fde392861e6404e8141a608f37de09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36071 } }
I20260923 21:00:46.940260 4552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:46.940352 4552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:46.940483 4552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:46.942778 4552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fde392861e6404e8141a608f37de09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36071 } }
I20260923 21:00:46.943259 4552 leader_election.cc:302] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fde392861e6404e8141a608f37de09c; no voters:
I20260923 21:00:46.943805 4552 leader_election.cc:288] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:46.944046 4557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:46.945477 4557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [term 1 LEADER]: Becoming Leader. State: Replica: 2fde392861e6404e8141a608f37de09c, State: Running, Role: LEADER
I20260923 21:00:46.946080 4552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:46.946085 4557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fde392861e6404e8141a608f37de09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36071 } }
I20260923 21:00:46.949267 4559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fde392861e6404e8141a608f37de09c. Latest consensus state: current_term: 1 leader_uuid: "2fde392861e6404e8141a608f37de09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fde392861e6404e8141a608f37de09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36071 } } }
I20260923 21:00:46.949709 4559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:46.950374 4558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fde392861e6404e8141a608f37de09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fde392861e6404e8141a608f37de09c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36071 } } }
I20260923 21:00:46.950830 4558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:46.952817 4566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:46.958954 4566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:46.969419 4566 catalog_manager.cc:1347] Generated new cluster ID: 96874d8ab34b422c9b646cdf8b0cdff6
I20260923 21:00:46.969570 4566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:46.995345 4566 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:46.997416 4566 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:47.014060 4566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c: Generated new TSK 0
I20260923 21:00:47.015316 4566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:47.072052 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36071
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:47.423920 4579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:47.424510 4579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:47.424715 4579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:47.432694 4579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:47.432873 4579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:47.433064 4579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:47.443516 4579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:47.445009 4579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:47.446573 4579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:47.461382 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:47.461345 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:47.461305 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:47.462888 4579 server_base.cc:1034] running on GCE node
I20260923 21:00:47.463686 4579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:47.464809 4579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:47.466087 4579 hybrid_clock.cc:648] HybridClock initialized: now 1790197247466009 us; error 67 us; skew 500 ppm
I20260923 21:00:47.466552 4579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:47 dist-test-slave-8xn5 krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:47.470172 4579 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:47.471550 4579 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:47.471704 4579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:47.472052 4579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:47.475407 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "3131002d83a144aeb5b79e547be01675"
format_stamp: "Formatted at 2026-09-23 21:00:47 on dist-test-slave-8xn5"
server_key: "586cc03967221e52fcb696b033e07483"
server_key_iv: "de462c2653ba6c76abcd7bbeed5e285b"
server_key_version: "encryptionkey@0"
I20260923 21:00:47.476329 4579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3131002d83a144aeb5b79e547be01675"
format_stamp: "Formatted at 2026-09-23 21:00:47 on dist-test-slave-8xn5"
server_key: "586cc03967221e52fcb696b033e07483"
server_key_iv: "de462c2653ba6c76abcd7bbeed5e285b"
server_key_version: "encryptionkey@0"
I20260923 21:00:47.482123 4579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 21:00:47.486485 4593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:47.488157 4579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:47.488472 4579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "3131002d83a144aeb5b79e547be01675"
format_stamp: "Formatted at 2026-09-23 21:00:47 on dist-test-slave-8xn5"
server_key: "586cc03967221e52fcb696b033e07483"
server_key_iv: "de462c2653ba6c76abcd7bbeed5e285b"
server_key_version: "encryptionkey@0"
I20260923 21:00:47.488740 4579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:47.507732 4579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:47.513015 4579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:47.513347 4579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:47.514484 4579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:47.516000 4579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:47.516134 4579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:47.516268 4579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:47.516358 4579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:47.551960 4579 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45747
I20260923 21:00:47.552016 4705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45747 every 8 connection(s)
I20260923 21:00:47.552974 4579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:47.555087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 4579
I20260923 21:00:47.555335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:47.556242 388 external_mini_cluster.cc:1428] Setting key 7246ea134d083478d69cbc9a19ca5ea9
Sep 23 21:00:47 dist-test-slave-8xn5 krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:00:47.578820 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:47.556365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49159 (local address 127.0.97.62:36071)
0923 21:00:47.556847 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:47.556851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:47.558393 (+ 1542us) server_negotiation.cc:407] Connection header received
0923 21:00:47.560240 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:47.560250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:47.560353 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:47.560491 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:47.562486 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:47.563849 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:47.566292 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:47.566838 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:47.566959 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:47.571975 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:47.571982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:47.571985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:47.572036 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:47.575349 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:47.576311 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:47.576316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:47.576319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:47.576453 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:47.576946 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:47.576951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:47.576953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:47.577288 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:47.577441 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:47.578309 (+ 868us) server_negotiation.cc:299] Negotiation successful
0923 21:00:47.578551 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260923 21:00:47.580171 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:47.556995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36071 (local address 127.0.97.1:49159)
0923 21:00:47.558003 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:47.558060 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:47.559911 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:47.560708 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:47.560717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:47.561341 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:47.562206 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:47.562238 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:47.564043 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:47.564050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:47.566082 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:47.566104 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:47.567117 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:47.567127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:47.567471 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:47.568422 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:47.568458 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:47.571650 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:47.575553 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:47.575569 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:47.575587 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:47.576154 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:47.576574 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:47.576578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:47.576587 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:47.576813 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:47.577423 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:47.577431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:47.577818 (+ 387us) client_negotiation.cc:769] Sending connection context
0923 21:00:47.578173 (+ 355us) client_negotiation.cc:240] Negotiation successful
0923 21:00:47.578715 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":365,"threads_started":1}
I20260923 21:00:47.582028 4706 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36071
I20260923 21:00:47.582448 4706 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:47.583190 4706 heartbeater.cc:511] Master 127.0.97.62:36071 requested a full tablet report, sending...
I20260923 21:00:47.585606 4517 ts_manager.cc:194] Registered new tserver with Master: 3131002d83a144aeb5b79e547be01675 (127.0.97.1:45747)
I20260923 21:00:47.587891 4517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:47.609802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36071
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:47.974740 4712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:47.975076 4712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:47.975246 4712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:47.983057 4712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:47.983213 4712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:47.983397 4712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:47.993921 4712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:47.995524 4712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:47.997114 4712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:48.012115 4718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:48.012144 4717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:48.012238 4720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:48.013183 4712 server_base.cc:1034] running on GCE node
I20260923 21:00:48.013885 4712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:48.015115 4712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:48.016419 4712 hybrid_clock.cc:648] HybridClock initialized: now 1790197248016365 us; error 56 us; skew 500 ppm
I20260923 21:00:48.016855 4712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:48 dist-test-slave-8xn5 krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:48.028409 4712 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:48.029801 4712 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:48.029956 4712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:48.030299 4712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:48.033934 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "5f351e270ceb4e6390aaf433280b5209"
format_stamp: "Formatted at 2026-09-23 21:00:48 on dist-test-slave-8xn5"
server_key: "b058ed6d42a9d6a8eecd8fca6ee883b0"
server_key_iv: "4d51cfff4661f58400af659d7b7b7ce0"
server_key_version: "encryptionkey@0"
I20260923 21:00:48.035172 4712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5f351e270ceb4e6390aaf433280b5209"
format_stamp: "Formatted at 2026-09-23 21:00:48 on dist-test-slave-8xn5"
server_key: "b058ed6d42a9d6a8eecd8fca6ee883b0"
server_key_iv: "4d51cfff4661f58400af659d7b7b7ce0"
server_key_version: "encryptionkey@0"
I20260923 21:00:48.041450 4712 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260923 21:00:48.045655 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:48.047731 4712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:48.047951 4712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "5f351e270ceb4e6390aaf433280b5209"
format_stamp: "Formatted at 2026-09-23 21:00:48 on dist-test-slave-8xn5"
server_key: "b058ed6d42a9d6a8eecd8fca6ee883b0"
server_key_iv: "4d51cfff4661f58400af659d7b7b7ce0"
server_key_version: "encryptionkey@0"
I20260923 21:00:48.048205 4712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:48.075105 4712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:48.080265 4712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:48.080593 4712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:48.081687 4712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:48.083334 4712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:48.083473 4712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:48.083611 4712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:48.083704 4712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:48.119930 4712 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42391
I20260923 21:00:48.119987 4838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42391 every 8 connection(s)
I20260923 21:00:48.120879 4712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:48.122217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 4712
I20260923 21:00:48.122476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:48.123224 388 external_mini_cluster.cc:1428] Setting key 9a72c7476883fc82c4e7a5e044c2a99a
Sep 23 21:00:48 dist-test-slave-8xn5 krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:00:48.145030 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.124225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38111 (local address 127.0.97.62:36071)
0923 21:00:48.124794 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.124800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.126373 (+ 1573us) server_negotiation.cc:407] Connection header received
0923 21:00:48.128106 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.128112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.128214 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.128341 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:48.130134 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.130984 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.132790 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.133570 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.133727 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.137780 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:48.137790 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:48.137799 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:48.137856 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:48.141587 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:48.142636 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:48.142643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:48.142648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:48.142843 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:48.143317 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:48.143320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:48.143322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:48.143640 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:48.143796 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.144503 (+ 707us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.144673 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":286,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:00:48.146374 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.124888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36071 (local address 127.0.97.2:38111)
0923 21:00:48.125813 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:48.125866 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:48.127779 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:48.128633 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:48.128644 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:48.129274 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:48.129909 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.129929 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.131163 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.131169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.132589 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.132605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.133833 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.133839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.134170 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.135009 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:48.135036 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:48.137438 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:48.141799 (+ 4361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:48.141808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:48.141824 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:48.142483 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:48.142948 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:48.142951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:48.142954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:48.143209 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:48.143754 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:48.143761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:48.144109 (+ 348us) client_negotiation.cc:769] Sending connection context
0923 21:00:48.144417 (+ 308us) client_negotiation.cc:240] Negotiation successful
0923 21:00:48.144931 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":398,"threads_started":1}
I20260923 21:00:48.148114 4839 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36071
I20260923 21:00:48.148481 4839 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:48.149339 4839 heartbeater.cc:511] Master 127.0.97.62:36071 requested a full tablet report, sending...
I20260923 21:00:48.151182 4517 ts_manager.cc:194] Registered new tserver with Master: 5f351e270ceb4e6390aaf433280b5209 (127.0.97.2:42391)
I20260923 21:00:48.152374 4517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:48.175747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36071
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:48.527446 4845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:48.527832 4845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:48.527998 4845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:48.536262 4845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:48.536468 4845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:48.536659 4845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:48.547312 4845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:48.548792 4845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:48.550261 4845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:48.566466 4850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:48.566556 4851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:48.566516 4853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:48.567659 4845 server_base.cc:1034] running on GCE node
I20260923 21:00:48.568282 4845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:48.569267 4845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:48.570487 4845 hybrid_clock.cc:648] HybridClock initialized: now 1790197248570433 us; error 41 us; skew 500 ppm
I20260923 21:00:48.570817 4845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:48 dist-test-slave-8xn5 krb5kdc[4473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:48.574440 4845 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:00:48.575790 4845 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:48.575915 4845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:48.576272 4845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:48.579581 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "e12c7b594d034c50b7e2d5787f42fee7"
format_stamp: "Formatted at 2026-09-23 21:00:48 on dist-test-slave-8xn5"
server_key: "a7efd75a1475f3fa0b3b845aec8cd0fe"
server_key_iv: "13018b15ace2c054403e2adbf51a7371"
server_key_version: "encryptionkey@0"
I20260923 21:00:48.580684 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e12c7b594d034c50b7e2d5787f42fee7"
format_stamp: "Formatted at 2026-09-23 21:00:48 on dist-test-slave-8xn5"
server_key: "a7efd75a1475f3fa0b3b845aec8cd0fe"
server_key_iv: "13018b15ace2c054403e2adbf51a7371"
server_key_version: "encryptionkey@0"
I20260923 21:00:48.586570 4845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 21:00:48.590914 4859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:48.592229 4706 heartbeater.cc:503] Master 127.0.97.62:36071 was elected leader, sending a full tablet report...
I20260923 21:00:48.592471 4845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:00:48.592670 4845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "e12c7b594d034c50b7e2d5787f42fee7"
format_stamp: "Formatted at 2026-09-23 21:00:48 on dist-test-slave-8xn5"
server_key: "a7efd75a1475f3fa0b3b845aec8cd0fe"
server_key_iv: "13018b15ace2c054403e2adbf51a7371"
server_key_version: "encryptionkey@0"
I20260923 21:00:48.592945 4845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:48.607847 4845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:48.612654 4845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:48.613004 4845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:48.614208 4845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:48.615804 4845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:48.615931 4845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:48.616076 4845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:48.616163 4845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:48.653276 4845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45187
I20260923 21:00:48.653337 4971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45187 every 8 connection(s)
I20260923 21:00:48.654277 4845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:48.657289 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 4845
I20260923 21:00:48.657519 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:48.657967 388 external_mini_cluster.cc:1428] Setting key 8dc5fd703e5fd9d02111ae70c6a6fad4
Sep 23 21:00:48 dist-test-slave-8xn5 krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:48.676651 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.657041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35761 (local address 127.0.97.62:36071)
0923 21:00:48.657609 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.657615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.659054 (+ 1439us) server_negotiation.cc:407] Connection header received
0923 21:00:48.661010 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.661015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.661125 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.661247 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:48.662968 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.663760 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.665426 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.665845 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.665995 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.669977 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:48.669983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:48.669986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:48.670023 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:48.673586 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:48.674532 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:48.674536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:48.674538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:48.674690 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:48.675063 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:48.675066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:48.675068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:48.675335 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:48.675455 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.676151 (+ 696us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.676359 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":299,"threads_started":1}
I20260923 21:00:48.677892 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.657833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36071 (local address 127.0.97.3:35761)
0923 21:00:48.658879 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:48.658937 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:48.660722 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:48.661514 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:48.661525 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:48.662107 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:48.662791 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.662810 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.663943 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.663946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.665252 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.665264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.666119 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.666126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.666424 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.667120 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:48.667148 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:48.669712 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:48.673762 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:48.673770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:48.673787 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:48.674413 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:48.674781 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:48.674784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:48.674787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:48.674967 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:48.675449 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:48.675454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:48.675808 (+ 354us) client_negotiation.cc:769] Sending connection context
0923 21:00:48.676048 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 21:00:48.676546 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":538,"threads_started":1}
I20260923 21:00:48.679421 4972 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36071
I20260923 21:00:48.679713 4972 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:48.680363 4972 heartbeater.cc:511] Master 127.0.97.62:36071 requested a full tablet report, sending...
I20260923 21:00:48.681785 4517 ts_manager.cc:194] Registered new tserver with Master: e12c7b594d034c50b7e2d5787f42fee7 (127.0.97.3:45187)
I20260923 21:00:48.682749 4517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35761
I20260923 21:00:48.686534 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:00:48.705156 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.691414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.0.97.62:36071)
0923 21:00:48.691633 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.691638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.692156 (+ 518us) server_negotiation.cc:407] Connection header received
0923 21:00:48.692327 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.692331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.692403 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.692495 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:48.694222 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.695025 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.696300 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.696780 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.696936 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.699483 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:48.699488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:48.699492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:48.699536 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:48.702383 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:48.703088 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:48.703093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:48.703095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:48.703320 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:48.703645 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:48.703648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:48.703650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:48.703967 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:48.704238 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.704589 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.704780 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 21:00:48.709797 4517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:48.717026 4517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:48.740342 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.731000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.0.97.1:45747)
0923 21:00:48.731718 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.731725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.731896 (+ 171us) server_negotiation.cc:407] Connection header received
0923 21:00:48.732174 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.732181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.732468 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.732726 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:48.733772 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.735638 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.737681 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.739084 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.739243 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.739499 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.739795 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.739966 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":390,"threads_started":1}
I20260923 21:00:48.740357 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.730730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45747 (local address 127.0.0.1:50790)
0923 21:00:48.731749 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:48.731786 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:48.731974 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:48.733028 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:48.733036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:48.733066 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:48.733549 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.733564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.735786 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.735794 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.737476 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.737490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.739300 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.739306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.739546 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.739594 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 21:00:48.739752 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 21:00:48.739979 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":646,"threads_started":1}
I20260923 21:00:48.742853 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.732636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.0.97.3:45187)
0923 21:00:48.734247 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.734256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.734290 (+ 34us) server_negotiation.cc:407] Connection header received
0923 21:00:48.734418 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.734427 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.734690 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.735525 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:48.735833 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.737498 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.740331 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.741658 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.741828 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.742158 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.742361 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.742526 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1447,"thread_start_us":1280,"threads_started":1}
I20260923 21:00:48.742954 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.732212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45187 (local address 127.0.0.1:58964)
0923 21:00:48.732746 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:48.732764 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:48.732916 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:48.734925 (+ 2009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:48.734930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:48.734955 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:48.735342 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.735353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.737674 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.737679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.740129 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.740144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.741871 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.741877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.742122 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.742163 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 21:00:48.742284 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 21:00:48.742488 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":1,"thread_start_us":276,"threads_started":1}
I20260923 21:00:48.743789 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.733726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42391 (local address 127.0.0.1:33458)
0923 21:00:48.734588 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:48.734604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:48.734795 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:48.735518 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:48.735522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:48.735544 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:48.735963 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.735977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.737829 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.737835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.740741 (+ 2906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.740754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.742954 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.742960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.743126 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.743163 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 21:00:48.743302 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 21:00:48.743433 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":603,"threads_started":1}
I20260923 21:00:48.743815 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.733086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.0.97.2:42391)
0923 21:00:48.733734 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.733743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.734698 (+ 955us) server_negotiation.cc:407] Connection header received
0923 21:00:48.734972 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.734979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.735220 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.735396 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:48.736147 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.737657 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.740967 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.742428 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.742573 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.742793 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.743364 (+ 571us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.743530 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":282,"threads_started":1}
I20260923 21:00:48.745764 4641 tablet_service.cc:1467] Processing CreateTablet for tablet e93e91ece56c4bb8b31cf87f6603391f (DEFAULT_TABLE table=test-table [id=3964593e8b6d4c5eaf90f9d73da2a61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:48.747394 4641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93e91ece56c4bb8b31cf87f6603391f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:48.746866 4907 tablet_service.cc:1467] Processing CreateTablet for tablet e93e91ece56c4bb8b31cf87f6603391f (DEFAULT_TABLE table=test-table [id=3964593e8b6d4c5eaf90f9d73da2a61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:48.747648 4774 tablet_service.cc:1467] Processing CreateTablet for tablet e93e91ece56c4bb8b31cf87f6603391f (DEFAULT_TABLE table=test-table [id=3964593e8b6d4c5eaf90f9d73da2a61e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:48.748281 4907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93e91ece56c4bb8b31cf87f6603391f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:48.748595 4774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93e91ece56c4bb8b31cf87f6603391f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:48.763312 4991 tablet_bootstrap.cc:492] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Bootstrap starting.
I20260923 21:00:48.764074 4992 tablet_bootstrap.cc:492] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Bootstrap starting.
I20260923 21:00:48.765071 4993 tablet_bootstrap.cc:492] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Bootstrap starting.
I20260923 21:00:48.767632 4991 tablet_bootstrap.cc:654] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:48.768824 4993 tablet_bootstrap.cc:654] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:48.768998 4992 tablet_bootstrap.cc:654] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:48.769070 4991 log.cc:824] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:48.770223 4992 log.cc:824] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:48.770550 4993 log.cc:824] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:48.774266 4992 tablet_bootstrap.cc:492] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: No bootstrap required, opened a new log
I20260923 21:00:48.774389 4991 tablet_bootstrap.cc:492] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: No bootstrap required, opened a new log
I20260923 21:00:48.774389 4993 tablet_bootstrap.cc:492] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: No bootstrap required, opened a new log
I20260923 21:00:48.774611 4992 ts_tablet_manager.cc:1397] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260923 21:00:48.774663 4993 ts_tablet_manager.cc:1397] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260923 21:00:48.774674 4991 ts_tablet_manager.cc:1397] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260923 21:00:48.782563 4992 raft_consensus.cc:348] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.783000 4992 raft_consensus.cc:374] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:48.783095 4992 raft_consensus.cc:729] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12c7b594d034c50b7e2d5787f42fee7, State: Initialized, Role: FOLLOWER
I20260923 21:00:48.783778 4992 consensus_queue.cc:260] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.784991 4972 heartbeater.cc:503] Master 127.0.97.62:36071 was elected leader, sending a full tablet report...
I20260923 21:00:48.785784 4992 ts_tablet_manager.cc:1428] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260923 21:00:48.785885 4993 raft_consensus.cc:348] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.785885 4991 raft_consensus.cc:348] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.786417 4993 raft_consensus.cc:374] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:48.786417 4991 raft_consensus.cc:374] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:48.786535 4993 raft_consensus.cc:729] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3131002d83a144aeb5b79e547be01675, State: Initialized, Role: FOLLOWER
I20260923 21:00:48.786535 4991 raft_consensus.cc:729] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f351e270ceb4e6390aaf433280b5209, State: Initialized, Role: FOLLOWER
I20260923 21:00:48.787232 4993 consensus_queue.cc:260] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.787220 4991 consensus_queue.cc:260] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.788441 4839 heartbeater.cc:503] Master 127.0.97.62:36071 was elected leader, sending a full tablet report...
I20260923 21:00:48.789590 4991 ts_tablet_manager.cc:1428] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260923 21:00:48.792781 4993 ts_tablet_manager.cc:1428] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
W20260923 21:00:48.805522 4707 tablet.cc:2369] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:00:48.872979 4840 tablet.cc:2369] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:00:48.876564 4997 raft_consensus.cc:482] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:48.876967 4997 raft_consensus.cc:504] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.878829 4997 leader_election.cc:288] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3131002d83a144aeb5b79e547be01675 (127.0.97.1:45747), 5f351e270ceb4e6390aaf433280b5209 (127.0.97.2:42391)
I20260923 21:00:48.884963 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.879243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40801 (local address 127.0.97.1:45747)
0923 21:00:48.879916 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.879922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.879943 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:00:48.880015 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.880022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.880101 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.880202 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:48.880700 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.881408 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.883155 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.884041 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.884211 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.884314 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.884568 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.884703 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
I20260923 21:00:48.885054 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.879240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45747 (local address 127.0.97.3:40801)
0923 21:00:48.879447 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:48.879465 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:48.879613 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:48.880331 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:48.880335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:48.880366 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:48.880556 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.880563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.881536 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.881544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.882978 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.882990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.884220 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.884225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.884414 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.884437 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 21:00:48.884542 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 21:00:48.884693 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":41,"spinlock_wait_cycles":36736}
I20260923 21:00:48.885353 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.879954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33893 (local address 127.0.97.2:42391)
0923 21:00:48.880238 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.880244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.880383 (+ 139us) server_negotiation.cc:407] Connection header received
0923 21:00:48.880588 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.880592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.880651 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.880740 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:00:48.881217 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.881921 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.883662 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.884438 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.884597 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.884691 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.884958 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.885088 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:00:48.885476 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.879626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42391 (local address 127.0.97.3:33893)
0923 21:00:48.880279 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:48.880294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:48.880453 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:48.880754 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:48.880758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:48.880780 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:00:48.881081 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.881088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.882066 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.882071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.883512 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:48.883523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.884652 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.884657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:48.884806 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.884827 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 21:00:48.884935 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 21:00:48.885067 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":1,"thread_start_us":380,"threads_started":1}
I20260923 21:00:48.886229 4661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e91ece56c4bb8b31cf87f6603391f" candidate_uuid: "e12c7b594d034c50b7e2d5787f42fee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3131002d83a144aeb5b79e547be01675" is_pre_election: true
I20260923 21:00:48.886559 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e91ece56c4bb8b31cf87f6603391f" candidate_uuid: "e12c7b594d034c50b7e2d5787f42fee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f351e270ceb4e6390aaf433280b5209" is_pre_election: true
I20260923 21:00:48.886812 4661 raft_consensus.cc:2454] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e12c7b594d034c50b7e2d5787f42fee7 in term 0.
I20260923 21:00:48.887080 4794 raft_consensus.cc:2454] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e12c7b594d034c50b7e2d5787f42fee7 in term 0.
I20260923 21:00:48.887535 4862 leader_election.cc:302] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3131002d83a144aeb5b79e547be01675, e12c7b594d034c50b7e2d5787f42fee7; no voters:
I20260923 21:00:48.888068 4997 raft_consensus.cc:2781] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:48.888343 4997 raft_consensus.cc:482] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:48.888475 4997 raft_consensus.cc:3037] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:48.890862 4997 raft_consensus.cc:504] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.891494 4997 leader_election.cc:288] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [CANDIDATE]: Term 1 election: Requested vote from peers 3131002d83a144aeb5b79e547be01675 (127.0.97.1:45747), 5f351e270ceb4e6390aaf433280b5209 (127.0.97.2:42391)
I20260923 21:00:48.892082 4661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e91ece56c4bb8b31cf87f6603391f" candidate_uuid: "e12c7b594d034c50b7e2d5787f42fee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3131002d83a144aeb5b79e547be01675"
I20260923 21:00:48.892243 4794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e91ece56c4bb8b31cf87f6603391f" candidate_uuid: "e12c7b594d034c50b7e2d5787f42fee7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f351e270ceb4e6390aaf433280b5209"
I20260923 21:00:48.892324 4661 raft_consensus.cc:3037] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:48.892489 4794 raft_consensus.cc:3037] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:48.895167 4661 raft_consensus.cc:2454] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12c7b594d034c50b7e2d5787f42fee7 in term 1.
I20260923 21:00:48.895334 4794 raft_consensus.cc:2454] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e12c7b594d034c50b7e2d5787f42fee7 in term 1.
I20260923 21:00:48.895685 4862 leader_election.cc:302] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3131002d83a144aeb5b79e547be01675, e12c7b594d034c50b7e2d5787f42fee7; no voters:
I20260923 21:00:48.896009 4997 raft_consensus.cc:2781] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:48.896698 4997 raft_consensus.cc:686] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 1 LEADER]: Becoming Leader. State: Replica: e12c7b594d034c50b7e2d5787f42fee7, State: Running, Role: LEADER
I20260923 21:00:48.897346 4997 consensus_queue.cc:237] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } }
I20260923 21:00:48.902613 4517 catalog_manager.cc:5495] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e12c7b594d034c50b7e2d5787f42fee7 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e12c7b594d034c50b7e2d5787f42fee7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e12c7b594d034c50b7e2d5787f42fee7" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 } health_report { overall_health: UNKNOWN } } }
W20260923 21:00:48.905958 4973 tablet.cc:2369] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:00:48.946383 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.940435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.0.97.3:45187)
0923 21:00:48.940688 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.940695 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:48.940752 (+ 57us) server_negotiation.cc:407] Connection header received
0923 21:00:48.940989 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:48.940994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:48.941074 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:48.941265 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:00:48.941588 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.942380 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.943979 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:48.944424 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:48.944531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:48.945104 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:00:48.945685 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:00:48.945900 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:48.945997 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 21:00:48.946071 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260923 21:00:48.956151 4661 raft_consensus.cc:1261] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 1 FOLLOWER]: Refusing update from remote peer e12c7b594d034c50b7e2d5787f42fee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:48.956230 4794 raft_consensus.cc:1261] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 1 FOLLOWER]: Refusing update from remote peer e12c7b594d034c50b7e2d5787f42fee7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:48.957312 5001 consensus_queue.cc:1035] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f351e270ceb4e6390aaf433280b5209" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:48.957711 4997 consensus_queue.cc:1035] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3131002d83a144aeb5b79e547be01675" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:48.977931 5006 mvcc.cc:205] Tried to move back new op lower bound from 7332647931712204800 to 7332647931488190464. Current Snapshot: MvccSnapshot[applied={T|T < 7332647931712204800}]
I20260923 21:00:48.988909 4517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:48.989346 4517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:48.993408 4517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fde392861e6404e8141a608f37de09c: Sending DeleteTablet for 3 replicas of tablet e93e91ece56c4bb8b31cf87f6603391f
I20260923 21:00:48.994845 4641 tablet_service.cc:1514] Processing DeleteTablet for tablet e93e91ece56c4bb8b31cf87f6603391f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50790
I20260923 21:00:48.995054 4774 tablet_service.cc:1514] Processing DeleteTablet for tablet e93e91ece56c4bb8b31cf87f6603391f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33458
I20260923 21:00:48.995064 4907 tablet_service.cc:1514] Processing DeleteTablet for tablet e93e91ece56c4bb8b31cf87f6603391f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58964
I20260923 21:00:48.997712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 4579
I20260923 21:00:49.006304 5016 tablet_replica.cc:330] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: stopping tablet replica
I20260923 21:00:49.007069 5016 raft_consensus.cc:2229] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:49.008055 5016 raft_consensus.cc:2258] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:49.010730 5015 tablet_replica.cc:330] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: stopping tablet replica
I20260923 21:00:49.011390 5015 raft_consensus.cc:2229] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:49.012001 5015 raft_consensus.cc:2258] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:49.013995 5016 ts_tablet_manager.cc:1905] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:49.014309 5015 ts_tablet_manager.cc:1905] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:49.016670 5014 tablet_replica.cc:330] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: stopping tablet replica
I20260923 21:00:49.017184 5014 raft_consensus.cc:2229] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:49.017724 5014 raft_consensus.cc:2258] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:49.021790 5015 ts_tablet_manager.cc:1918] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:49.021991 5015 log.cc:1196] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/e93e91ece56c4bb8b31cf87f6603391f
I20260923 21:00:49.022600 5015 ts_tablet_manager.cc:1939] T e93e91ece56c4bb8b31cf87f6603391f P 5f351e270ceb4e6390aaf433280b5209: Deleting consensus metadata
I20260923 21:00:49.023254 5014 ts_tablet_manager.cc:1905] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:49.023314 5016 ts_tablet_manager.cc:1918] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:49.023648 5016 log.cc:1196] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/e93e91ece56c4bb8b31cf87f6603391f
I20260923 21:00:49.024323 5016 ts_tablet_manager.cc:1939] T e93e91ece56c4bb8b31cf87f6603391f P e12c7b594d034c50b7e2d5787f42fee7: Deleting consensus metadata
I20260923 21:00:49.024788 4503 catalog_manager.cc:4841] TS 5f351e270ceb4e6390aaf433280b5209 (127.0.97.2:42391): tablet e93e91ece56c4bb8b31cf87f6603391f (table test-table [id=3964593e8b6d4c5eaf90f9d73da2a61e]) successfully deleted
I20260923 21:00:49.026252 4503 catalog_manager.cc:4841] TS e12c7b594d034c50b7e2d5787f42fee7 (127.0.97.3:45187): tablet e93e91ece56c4bb8b31cf87f6603391f (table test-table [id=3964593e8b6d4c5eaf90f9d73da2a61e]) successfully deleted
I20260923 21:00:49.030222 5014 ts_tablet_manager.cc:1918] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:49.030449 5014 log.cc:1196] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/e93e91ece56c4bb8b31cf87f6603391f
Sep 23 21:00:49 dist-test-slave-8xn5 krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:00:49.031009 5014 ts_tablet_manager.cc:1939] T e93e91ece56c4bb8b31cf87f6603391f P 3131002d83a144aeb5b79e547be01675: Deleting consensus metadata
I20260923 21:00:49.032974 4503 catalog_manager.cc:4841] TS 3131002d83a144aeb5b79e547be01675 (127.0.97.1:45747): tablet e93e91ece56c4bb8b31cf87f6603391f (table test-table [id=3964593e8b6d4c5eaf90f9d73da2a61e]) successfully deleted
I20260923 21:00:49.038450 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:48.999184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.0.97.1:45747)
0923 21:00:48.999387 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:48.999393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:49.011959 (+ 12566us) server_negotiation.cc:407] Connection header received
0923 21:00:49.012512 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:49.012522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:49.012656 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:49.012742 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:49.015038 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:49.016198 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:49.020216 (+ 4018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:49.020873 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:49.020980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:49.032048 (+ 11068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:49.032060 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:49.032078 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:49.032138 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:49.035320 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:49.036329 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:49.036336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:49.036338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:49.036478 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:49.036880 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:49.036883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:49.036885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:49.037246 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:49.037467 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:49.037826 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 21:00:49.037990 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 21:00:49.039306 4701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:49.136508 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 4579
I20260923 21:00:49.159828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 4712
Sep 23 21:00:49 dist-test-slave-8xn5 krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:00:49.176546 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:49.160756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.97.2:42391)
0923 21:00:49.160933 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:49.160938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:49.160958 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:00:49.161063 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:49.161068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:49.161152 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:49.161258 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:49.162589 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:49.163398 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:49.164657 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:49.165077 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:49.165213 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:49.170178 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:49.170197 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:49.170214 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:49.170258 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:49.173532 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:49.174515 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:49.174524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:49.174526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:49.174657 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:49.175070 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:49.175073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:49.175075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:49.175449 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:49.175676 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:49.176004 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 21:00:49.176178 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 21:00:49.177326 4834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:49.271636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 4712
I20260923 21:00:49.288591 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 4845
Sep 23 21:00:49 dist-test-slave-8xn5 krb5kdc[4473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:00:49.307364 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:49.289455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.0.97.3:45187)
0923 21:00:49.289665 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:49.289673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:49.289711 (+ 38us) server_negotiation.cc:407] Connection header received
0923 21:00:49.289854 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:49.289860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:49.289963 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:49.290079 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:49.291393 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:49.292602 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:49.293979 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:49.294728 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:49.294884 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:49.299663 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:49.299685 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:49.299717 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:49.299790 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:49.304123 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:49.304954 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:49.304965 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:49.304969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:49.305154 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:49.305522 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:49.305528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:49.305531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:49.306034 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:49.306389 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:49.306682 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 21:00:49.306934 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 21:00:49.308123 4967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:49.433950 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 4845
I20260923 21:00:49.453620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 4486
I20260923 21:00:49.454857 4547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:49.569331 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 4486
2026-09-23T21:00:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4565 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T21:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:49Z Disabled control of system clock
I20260923 21:00:49.642851 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38759
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 21:00:49.991850 5025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:49.992201 5025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:49.992293 5025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:50.000612 5025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:50.000792 5025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:50.000876 5025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:50.000960 5025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:50.001021 5025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:50.010946 5025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:50.012339 5025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:50.014405 5025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:50.027838 5030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:50.027854 5031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:50.028231 5033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:50.029016 5025 server_base.cc:1034] running on GCE node
I20260923 21:00:50.030017 5025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:50.031679 5025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:50.032985 5025 hybrid_clock.cc:648] HybridClock initialized: now 1790197250032919 us; error 60 us; skew 500 ppm
I20260923 21:00:50.033454 5025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:50.036401 5025 webserver.cc:460] Webserver started at http://127.0.97.62:34247/ using document root <none> and password file <none>
I20260923 21:00:50.037302 5025 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:50.037464 5025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:50.037848 5025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:50.041388 5025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "2d34ecc18b624e39bcd3568f78624485"
format_stamp: "Formatted at 2026-09-23 21:00:50 on dist-test-slave-8xn5"
server_key: "cfeb4fd5017a4ae38715abf90bd788be"
server_key_iv: "b545b607c4c49af6cd347da1f531de68"
server_key_version: "encryptionkey@0"
I20260923 21:00:50.042598 5025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "2d34ecc18b624e39bcd3568f78624485"
format_stamp: "Formatted at 2026-09-23 21:00:50 on dist-test-slave-8xn5"
server_key: "cfeb4fd5017a4ae38715abf90bd788be"
server_key_iv: "b545b607c4c49af6cd347da1f531de68"
server_key_version: "encryptionkey@0"
I20260923 21:00:50.048856 5025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 21:00:50.053571 5039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:50.055338 5025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:00:50.055604 5025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "2d34ecc18b624e39bcd3568f78624485"
format_stamp: "Formatted at 2026-09-23 21:00:50 on dist-test-slave-8xn5"
server_key: "cfeb4fd5017a4ae38715abf90bd788be"
server_key_iv: "b545b607c4c49af6cd347da1f531de68"
server_key_version: "encryptionkey@0"
I20260923 21:00:50.055877 5025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:50.063160 5025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:50.064153 5025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:50.064510 5025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:50.085677 5025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38759
I20260923 21:00:50.085691 5090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38759 every 8 connection(s)
I20260923 21:00:50.087284 5025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:50.091594 5091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:50.095043 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5025
I20260923 21:00:50.095266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:50.096067 388 external_mini_cluster.cc:1428] Setting key e5c165ff2b5060c9ad3f81d321fda294
I20260923 21:00:50.105602 5091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485: Bootstrap starting.
I20260923 21:00:50.105772 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:50.098651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.0.97.62:38759)
0923 21:00:50.099860 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:50.099872 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:50.099918 (+ 46us) server_negotiation.cc:407] Connection header received
0923 21:00:50.101638 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:50.101669 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:50.102212 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:50.102779 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:50.103992 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:50.103998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:50.104013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:50.104027 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:50.104100 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:50.104116 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:50.104354 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:50.104493 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 21:00:50.104769 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":489,"threads_started":1}
I20260923 21:00:50.109464 5091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:50.110682 5091 log.cc:824] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:50.114471 5091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485: No bootstrap required, opened a new log
I20260923 21:00:50.124259 5091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d34ecc18b624e39bcd3568f78624485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38759 } }
I20260923 21:00:50.124794 5091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:50.124928 5091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d34ecc18b624e39bcd3568f78624485, State: Initialized, Role: FOLLOWER
I20260923 21:00:50.125763 5091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d34ecc18b624e39bcd3568f78624485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38759 } }
I20260923 21:00:50.126194 5091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:50.126439 5091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:50.126677 5091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:50.129366 5091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d34ecc18b624e39bcd3568f78624485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38759 } }
I20260923 21:00:50.130048 5091 leader_election.cc:302] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d34ecc18b624e39bcd3568f78624485; no voters:
I20260923 21:00:50.130823 5091 leader_election.cc:288] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:50.130971 5096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:50.131829 5096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [term 1 LEADER]: Becoming Leader. State: Replica: 2d34ecc18b624e39bcd3568f78624485, State: Running, Role: LEADER
I20260923 21:00:50.132493 5096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d34ecc18b624e39bcd3568f78624485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38759 } }
I20260923 21:00:50.133669 5091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:50.135375 5098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d34ecc18b624e39bcd3568f78624485. Latest consensus state: current_term: 1 leader_uuid: "2d34ecc18b624e39bcd3568f78624485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d34ecc18b624e39bcd3568f78624485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38759 } } }
I20260923 21:00:50.135776 5098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:50.136611 5097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d34ecc18b624e39bcd3568f78624485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d34ecc18b624e39bcd3568f78624485" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38759 } } }
I20260923 21:00:50.138049 5097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:50.138289 5101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:50.146844 5101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:50.157858 5101 catalog_manager.cc:1347] Generated new cluster ID: 944767aa4ba344b7858c5ccb78ffc433
I20260923 21:00:50.158118 5101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:50.174005 5101 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:50.174646 5101 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:50.185825 5101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485: Generated new TSK 0
I20260923 21:00:50.187124 5101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:00:50.197021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38759
--builtin_ntp_servers=127.0.97.20:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 21:00:50.561173 5115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:50.561553 5115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:50.561749 5115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:50.570034 5115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:50.570212 5115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:50.570426 5115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:50.585163 5115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:50.587031 5115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:50.589114 5115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:50.608006 5121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:50.608255 5120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:50.608006 5123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:50.608990 5115 server_base.cc:1034] running on GCE node
I20260923 21:00:50.610098 5115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:50.611714 5115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:50.612999 5115 hybrid_clock.cc:648] HybridClock initialized: now 1790197250612922 us; error 60 us; skew 500 ppm
I20260923 21:00:50.613510 5115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:50.617174 5115 webserver.cc:460] Webserver started at http://127.0.97.1:40953/ using document root <none> and password file <none>
I20260923 21:00:50.618364 5115 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:50.618517 5115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:50.618973 5115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:50.624004 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "31d43d0739a748ff8441d7fd4969fdf7"
format_stamp: "Formatted at 2026-09-23 21:00:50 on dist-test-slave-8xn5"
server_key: "e773e6b4f99581cb0439bc641d2b1290"
server_key_iv: "0957833605957d96c1d306615d8b1c0a"
server_key_version: "encryptionkey@0"
I20260923 21:00:50.625387 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "31d43d0739a748ff8441d7fd4969fdf7"
format_stamp: "Formatted at 2026-09-23 21:00:50 on dist-test-slave-8xn5"
server_key: "e773e6b4f99581cb0439bc641d2b1290"
server_key_iv: "0957833605957d96c1d306615d8b1c0a"
server_key_version: "encryptionkey@0"
I20260923 21:00:50.632643 5115 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260923 21:00:50.636914 5129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:50.638734 5115 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:50.638947 5115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "31d43d0739a748ff8441d7fd4969fdf7"
format_stamp: "Formatted at 2026-09-23 21:00:50 on dist-test-slave-8xn5"
server_key: "e773e6b4f99581cb0439bc641d2b1290"
server_key_iv: "0957833605957d96c1d306615d8b1c0a"
server_key_version: "encryptionkey@0"
I20260923 21:00:50.639216 5115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:50.645725 5115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:50.646871 5115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:50.647193 5115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:50.648222 5115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:50.649753 5115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:50.649879 5115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:50.650017 5115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:50.650108 5115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:50.686303 5115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37477
I20260923 21:00:50.686411 5241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37477 every 8 connection(s)
I20260923 21:00:50.687855 5115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:50.689719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5115
I20260923 21:00:50.689924 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:50.690696 388 external_mini_cluster.cc:1428] Setting key cd59cc9ed3bfabe12e13964e370138ba
I20260923 21:00:50.694133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38759
--builtin_ntp_servers=127.0.97.20:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 21:00:50.699479 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:50.691085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38919 (local address 127.0.97.62:38759)
0923 21:00:50.691585 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:50.691591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:50.693536 (+ 1945us) server_negotiation.cc:407] Connection header received
0923 21:00:50.696628 (+ 3092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:50.696633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:50.696732 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:50.696848 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:50.698700 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:50.698706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:50.698710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:50.698731 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:50.698781 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:50.698793 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:50.698906 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:50.699160 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 21:00:50.699219 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":253,"threads_started":1}
I20260923 21:00:50.701009 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:50.691926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38759 (local address 127.0.97.1:38919)
0923 21:00:50.693364 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:50.693415 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:50.696198 (+ 2783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:50.697220 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:50.697236 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:50.697400 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:50.698231 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:50.698236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:50.698297 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:50.698475 (+ 178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:50.698905 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:50.698918 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:50.698930 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 21:00:50.699258 (+ 328us) client_negotiation.cc:240] Negotiation successful
0923 21:00:50.699596 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":374,"threads_started":1}
I20260923 21:00:50.703001 5242 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38759
I20260923 21:00:50.703367 5242 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:50.704031 5242 heartbeater.cc:511] Master 127.0.97.62:38759 requested a full tablet report, sending...
I20260923 21:00:50.706543 5056 ts_manager.cc:194] Registered new tserver with Master: 31d43d0739a748ff8441d7fd4969fdf7 (127.0.97.1:37477)
W20260923 21:00:51.042066 5246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:51.042465 5246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:51.042626 5246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:51.050383 5246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:51.050551 5246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:51.050743 5246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:51.060617 5246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:51.062135 5246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:51.064070 5246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:51.079406 5251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:51.079394 5254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:51.079097 5252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:51.082003 5246 server_base.cc:1034] running on GCE node
I20260923 21:00:51.082859 5246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:51.083967 5246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:51.085246 5246 hybrid_clock.cc:648] HybridClock initialized: now 1790197251085182 us; error 53 us; skew 500 ppm
I20260923 21:00:51.085619 5246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:51.101799 5246 webserver.cc:460] Webserver started at http://127.0.97.2:46151/ using document root <none> and password file <none>
I20260923 21:00:51.102775 5246 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:51.102931 5246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:51.103307 5246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:51.106685 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "f8b4e2da53d943b18d66903c0b9c7e76"
format_stamp: "Formatted at 2026-09-23 21:00:51 on dist-test-slave-8xn5"
server_key: "3fa2c450a59bcc77e4891afa2ca19673"
server_key_iv: "8402fd800fff4c27d37ff3f12bdbe0bc"
server_key_version: "encryptionkey@0"
I20260923 21:00:51.107595 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f8b4e2da53d943b18d66903c0b9c7e76"
format_stamp: "Formatted at 2026-09-23 21:00:51 on dist-test-slave-8xn5"
server_key: "3fa2c450a59bcc77e4891afa2ca19673"
server_key_iv: "8402fd800fff4c27d37ff3f12bdbe0bc"
server_key_version: "encryptionkey@0"
I20260923 21:00:51.113535 5246 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 21:00:51.117728 5260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:51.119699 5246 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260923 21:00:51.119935 5246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "f8b4e2da53d943b18d66903c0b9c7e76"
format_stamp: "Formatted at 2026-09-23 21:00:51 on dist-test-slave-8xn5"
server_key: "3fa2c450a59bcc77e4891afa2ca19673"
server_key_iv: "8402fd800fff4c27d37ff3f12bdbe0bc"
server_key_version: "encryptionkey@0"
I20260923 21:00:51.120250 5246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:51.126858 5246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:51.127877 5246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:51.128186 5246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:51.129292 5246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:51.130927 5246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:51.131016 5246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:51.131170 5246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:51.131268 5246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:51.168910 5246 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40583
I20260923 21:00:51.168963 5372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40583 every 8 connection(s)
I20260923 21:00:51.170418 5246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:51.178930 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5246
I20260923 21:00:51.179204 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:51.179674 388 external_mini_cluster.cc:1428] Setting key 1588ee7a8fb1e65dcea330d0068bbc59
I20260923 21:00:51.180634 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.173406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46907 (local address 127.0.97.62:38759)
0923 21:00:51.173595 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.173600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.175525 (+ 1925us) server_negotiation.cc:407] Connection header received
0923 21:00:51.177478 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.177483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.177544 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.177657 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.179721 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.179727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.179731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.179752 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.179813 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.179833 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.180006 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.180236 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.180319 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 21:00:51.181802 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.174012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38759 (local address 127.0.97.2:46907)
0923 21:00:51.175279 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:51.175340 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:51.177113 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:51.177917 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:51.177927 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:51.178127 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:51.179083 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:51.179087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:51.179278 (+ 191us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:51.179435 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:51.179969 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:51.179983 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:51.179992 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 21:00:51.180334 (+ 342us) client_negotiation.cc:240] Negotiation successful
0923 21:00:51.180688 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":426,"threads_started":1}
I20260923 21:00:51.182682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38759
--builtin_ntp_servers=127.0.97.20:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 21:00:51.183490 5373 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38759
I20260923 21:00:51.183899 5373 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:51.184574 5373 heartbeater.cc:511] Master 127.0.97.62:38759 requested a full tablet report, sending...
I20260923 21:00:51.186215 5056 ts_manager.cc:194] Registered new tserver with Master: f8b4e2da53d943b18d66903c0b9c7e76 (127.0.97.2:40583)
W20260923 21:00:51.543274 5376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:51.543614 5376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:51.543782 5376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:51.552291 5376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:51.552446 5376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:51.552567 5376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:00:51.563787 5376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:51.565157 5376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:51.567123 5376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:51.582451 5381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:51.582674 5382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:51.582863 5384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:51.584048 5376 server_base.cc:1034] running on GCE node
I20260923 21:00:51.584839 5376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:51.586007 5376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:51.587306 5376 hybrid_clock.cc:648] HybridClock initialized: now 1790197251587191 us; error 110 us; skew 500 ppm
I20260923 21:00:51.587697 5376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:51.590499 5376 webserver.cc:460] Webserver started at http://127.0.97.3:38781/ using document root <none> and password file <none>
I20260923 21:00:51.591341 5376 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:51.591506 5376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:51.591866 5376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:51.595350 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "1e567a2310df4216aa950f1220ab4bda"
format_stamp: "Formatted at 2026-09-23 21:00:51 on dist-test-slave-8xn5"
server_key: "d2cea37b40cd32fba5fbed7bc8164925"
server_key_iv: "3e8f095e26066d1bfc517dd93564b1d7"
server_key_version: "encryptionkey@0"
I20260923 21:00:51.596386 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1e567a2310df4216aa950f1220ab4bda"
format_stamp: "Formatted at 2026-09-23 21:00:51 on dist-test-slave-8xn5"
server_key: "d2cea37b40cd32fba5fbed7bc8164925"
server_key_iv: "3e8f095e26066d1bfc517dd93564b1d7"
server_key_version: "encryptionkey@0"
I20260923 21:00:51.602837 5376 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:00:51.607165 5390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:51.608737 5376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 21:00:51.608958 5376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "1e567a2310df4216aa950f1220ab4bda"
format_stamp: "Formatted at 2026-09-23 21:00:51 on dist-test-slave-8xn5"
server_key: "d2cea37b40cd32fba5fbed7bc8164925"
server_key_iv: "3e8f095e26066d1bfc517dd93564b1d7"
server_key_version: "encryptionkey@0"
I20260923 21:00:51.609305 5376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:51.615842 5376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:51.616945 5376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:51.617268 5376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:51.618307 5376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:51.619982 5376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:51.620134 5376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:51.620276 5376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:51.620380 5376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:51.657635 5376 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40141
I20260923 21:00:51.657686 5502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40141 every 8 connection(s)
I20260923 21:00:51.659337 5376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:00:51.660367 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5376
I20260923 21:00:51.660532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:00:51.661118 388 external_mini_cluster.cc:1428] Setting key f8e489516ae718d18fd1c751e23c630f
I20260923 21:00:51.669775 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.663438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59585 (local address 127.0.97.62:38759)
0923 21:00:51.663643 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.663649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.665428 (+ 1779us) server_negotiation.cc:407] Connection header received
0923 21:00:51.667287 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.667292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.667355 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.667459 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.669093 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.669098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.669101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.669123 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.669159 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.669173 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.669302 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.669468 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.669535 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:00:51.670948 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.663989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38759 (local address 127.0.97.3:59585)
0923 21:00:51.665245 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:51.665292 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:51.666975 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:51.667742 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:51.667751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:51.667864 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:51.668726 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:51.668728 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:51.668773 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:51.668864 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:51.669280 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:51.669288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:51.669294 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 21:00:51.669520 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 21:00:51.669814 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":445,"threads_started":1}
I20260923 21:00:51.672799 5503 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38759
I20260923 21:00:51.673139 5503 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:51.673787 5503 heartbeater.cc:511] Master 127.0.97.62:38759 requested a full tablet report, sending...
I20260923 21:00:51.675557 5056 ts_manager.cc:194] Registered new tserver with Master: 1e567a2310df4216aa950f1220ab4bda (127.0.97.3:40141)
I20260923 21:00:51.677783 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:00:51.685567 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.682572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.0.97.62:38759)
0923 21:00:51.682766 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.682771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.683328 (+ 557us) server_negotiation.cc:407] Connection header received
0923 21:00:51.683528 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.683534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.683606 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.683724 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.684747 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.684752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.684756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.684775 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.684810 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.684824 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.685074 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.685117 (+ 43us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.685166 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:00:51.690270 5056 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:00:51.697330 5056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:00:51.709991 5242 heartbeater.cc:503] Master 127.0.97.62:38759 was elected leader, sending a full tablet report...
I20260923 21:00:51.716740 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.711440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37477 (local address 127.0.0.1:40920)
0923 21:00:51.712345 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:51.712390 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:51.712605 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:51.713746 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:51.713761 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:51.713894 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:51.714978 (+ 1084us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:51.714982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:51.715023 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:51.715114 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:51.715736 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:51.715748 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:51.715761 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 21:00:51.715995 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 21:00:51.716103 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":428,"threads_started":1}
I20260923 21:00:51.717957 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.711626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.0.97.1:37477)
0923 21:00:51.712294 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.712304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.712546 (+ 242us) server_negotiation.cc:407] Connection header received
0923 21:00:51.712771 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.712780 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.713054 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.713264 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.715354 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.715369 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.715390 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.715418 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.715495 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.715516 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.715676 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.717457 (+ 1781us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.717576 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":352,"threads_started":1}
I20260923 21:00:51.719157 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.713192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.0.97.3:40141)
0923 21:00:51.714170 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.714180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.714827 (+ 647us) server_negotiation.cc:407] Connection header received
0923 21:00:51.715129 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.715141 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.715368 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.715471 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.717942 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.717951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.717965 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.717986 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.718054 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.718081 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.718236 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.718562 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.718715 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":661,"threads_started":1}
I20260923 21:00:51.720084 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.715042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.0.97.2:40583)
0923 21:00:51.717498 (+ 2456us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.717511 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.717558 (+ 47us) server_negotiation.cc:407] Connection header received
0923 21:00:51.717774 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.717784 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.718060 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.718215 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.719258 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.719269 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.719284 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.719302 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.719359 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.719386 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.719559 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.719629 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.719741 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2280,"thread_start_us":2302,"threads_started":1}
I20260923 21:00:51.720134 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.715594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40583 (local address 127.0.0.1:59954)
0923 21:00:51.717462 (+ 1868us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:51.717483 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:51.717582 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:51.718310 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:51.718316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:51.718372 (+ 56us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:51.719058 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:51.719061 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:51.719077 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:51.719117 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:51.719507 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:51.719512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:51.719517 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 21:00:51.719642 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 21:00:51.719693 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1749,"thread_start_us":1600,"threads_started":1}
I20260923 21:00:51.722126 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.712834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40141 (local address 127.0.0.1:40758)
0923 21:00:51.714711 (+ 1877us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:51.714731 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:51.714921 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:51.716705 (+ 1784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:51.716714 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:51.716747 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:51.717713 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:51.717717 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:51.717736 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:51.717779 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:51.718367 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:51.718372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:51.718377 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 21:00:51.721591 (+ 3214us) client_negotiation.cc:240] Negotiation successful
0923 21:00:51.721672 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1711,"thread_start_us":1576,"threads_started":1}
I20260923 21:00:51.723240 5177 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 (DEFAULT_TABLE table=test-table [id=f78438752afa48839928bec7ed79cc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:51.724988 5177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:51.724261 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 (DEFAULT_TABLE table=test-table [id=f78438752afa48839928bec7ed79cc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:51.726402 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:51.726083 5438 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 (DEFAULT_TABLE table=test-table [id=f78438752afa48839928bec7ed79cc26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:00:51.727417 5438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:51.740155 5521 tablet_bootstrap.cc:492] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Bootstrap starting.
I20260923 21:00:51.742686 5522 tablet_bootstrap.cc:492] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Bootstrap starting.
I20260923 21:00:51.744827 5523 tablet_bootstrap.cc:492] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Bootstrap starting.
I20260923 21:00:51.746387 5521 tablet_bootstrap.cc:654] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:51.746549 5522 tablet_bootstrap.cc:654] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:51.747857 5521 log.cc:824] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:51.747856 5522 log.cc:824] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:51.748466 5523 tablet_bootstrap.cc:654] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:51.750008 5523 log.cc:824] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:51.751897 5522 tablet_bootstrap.cc:492] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: No bootstrap required, opened a new log
I20260923 21:00:51.752399 5522 ts_tablet_manager.cc:1397] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260923 21:00:51.752691 5521 tablet_bootstrap.cc:492] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: No bootstrap required, opened a new log
I20260923 21:00:51.752962 5521 ts_tablet_manager.cc:1397] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260923 21:00:51.753235 5523 tablet_bootstrap.cc:492] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: No bootstrap required, opened a new log
I20260923 21:00:51.753540 5523 ts_tablet_manager.cc:1397] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 21:00:51.762459 5521 raft_consensus.cc:348] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.762935 5521 raft_consensus.cc:374] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:51.763039 5521 raft_consensus.cc:729] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e567a2310df4216aa950f1220ab4bda, State: Initialized, Role: FOLLOWER
I20260923 21:00:51.763845 5521 consensus_queue.cc:260] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.764362 5523 raft_consensus.cc:348] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.765023 5523 raft_consensus.cc:374] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:51.765139 5523 raft_consensus.cc:729] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b4e2da53d943b18d66903c0b9c7e76, State: Initialized, Role: FOLLOWER
I20260923 21:00:51.765064 5522 raft_consensus.cc:348] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.765554 5522 raft_consensus.cc:374] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:51.765663 5522 raft_consensus.cc:729] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d43d0739a748ff8441d7fd4969fdf7, State: Initialized, Role: FOLLOWER
I20260923 21:00:51.766047 5523 consensus_queue.cc:260] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.766471 5522 consensus_queue.cc:260] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.766934 5503 heartbeater.cc:503] Master 127.0.97.62:38759 was elected leader, sending a full tablet report...
I20260923 21:00:51.768600 5522 ts_tablet_manager.cc:1428] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260923 21:00:51.769304 5521 ts_tablet_manager.cc:1428] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260923 21:00:51.770476 5373 heartbeater.cc:503] Master 127.0.97.62:38759 was elected leader, sending a full tablet report...
I20260923 21:00:51.771384 5523 ts_tablet_manager.cc:1428] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260923 21:00:51.830504 5527 raft_consensus.cc:482] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:00:51.830924 5527 raft_consensus.cc:504] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.832852 5527 leader_election.cc:288] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31d43d0739a748ff8441d7fd4969fdf7 (127.0.97.1:37477), f8b4e2da53d943b18d66903c0b9c7e76 (127.0.97.2:40583)
I20260923 21:00:51.836012 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.833512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37477 (local address 127.0.97.3:56823)
0923 21:00:51.833776 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:51.833794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:51.833888 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:51.834383 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:51.834389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:51.834414 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:51.835202 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:51.835206 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:51.835226 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:51.835278 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:51.835684 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:51.835687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:51.835690 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 21:00:51.835774 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 21:00:51.835820 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 21:00:51.836274 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.833612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56823 (local address 127.0.97.1:37477)
0923 21:00:51.833834 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.833842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.833871 (+ 29us) server_negotiation.cc:407] Connection header received
0923 21:00:51.834077 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.834084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.834160 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.834309 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.835449 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.835455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.835459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.835502 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.835547 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.835565 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.835724 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.835786 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.835876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 21:00:51.837504 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.834334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43843 (local address 127.0.97.2:40583)
0923 21:00:51.834549 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.834556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.835216 (+ 660us) server_negotiation.cc:407] Connection header received
0923 21:00:51.835349 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.835357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.835421 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.835506 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.836934 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.836939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.836942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.836958 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.836989 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.837000 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.837095 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.837259 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.837308 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:00:51.837510 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5bdf5c07104ac3a0d435e5e353d8f5" candidate_uuid: "1e567a2310df4216aa950f1220ab4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d43d0739a748ff8441d7fd4969fdf7" is_pre_election: true
I20260923 21:00:51.837898 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.834097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40583 (local address 127.0.97.3:43843)
0923 21:00:51.835102 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:51.835125 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:51.835246 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:51.835567 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:51.835573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:51.835603 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:51.836675 (+ 1072us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:00:51.836677 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:51.836691 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:00:51.836733 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:51.837119 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:51.837125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:51.837131 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 21:00:51.837282 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 21:00:51.837347 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":640,"threads_started":1}
I20260923 21:00:51.838279 5197 raft_consensus.cc:2454] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e567a2310df4216aa950f1220ab4bda in term 0.
I20260923 21:00:51.839018 5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5bdf5c07104ac3a0d435e5e353d8f5" candidate_uuid: "1e567a2310df4216aa950f1220ab4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" is_pre_election: true
I20260923 21:00:51.839259 5393 leader_election.cc:302] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e567a2310df4216aa950f1220ab4bda, 31d43d0739a748ff8441d7fd4969fdf7; no voters:
I20260923 21:00:51.839627 5328 raft_consensus.cc:2454] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e567a2310df4216aa950f1220ab4bda in term 0.
I20260923 21:00:51.839761 5527 raft_consensus.cc:2781] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:00:51.839888 5527 raft_consensus.cc:482] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:00:51.840049 5527 raft_consensus.cc:3037] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:51.842908 5527 raft_consensus.cc:504] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.843706 5527 leader_election.cc:288] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [CANDIDATE]: Term 1 election: Requested vote from peers 31d43d0739a748ff8441d7fd4969fdf7 (127.0.97.1:37477), f8b4e2da53d943b18d66903c0b9c7e76 (127.0.97.2:40583)
I20260923 21:00:51.844416 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5bdf5c07104ac3a0d435e5e353d8f5" candidate_uuid: "1e567a2310df4216aa950f1220ab4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d43d0739a748ff8441d7fd4969fdf7"
I20260923 21:00:51.844446 5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5bdf5c07104ac3a0d435e5e353d8f5" candidate_uuid: "1e567a2310df4216aa950f1220ab4bda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b4e2da53d943b18d66903c0b9c7e76"
I20260923 21:00:51.844650 5197 raft_consensus.cc:3037] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:51.844676 5328 raft_consensus.cc:3037] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:51.847143 5328 raft_consensus.cc:2454] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e567a2310df4216aa950f1220ab4bda in term 1.
I20260923 21:00:51.847441 5197 raft_consensus.cc:2454] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e567a2310df4216aa950f1220ab4bda in term 1.
I20260923 21:00:51.847699 5393 leader_election.cc:302] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e567a2310df4216aa950f1220ab4bda, f8b4e2da53d943b18d66903c0b9c7e76; no voters:
I20260923 21:00:51.848095 5527 raft_consensus.cc:2781] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:51.848609 5527 raft_consensus.cc:686] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 1 LEADER]: Becoming Leader. State: Replica: 1e567a2310df4216aa950f1220ab4bda, State: Running, Role: LEADER
I20260923 21:00:51.849121 5527 consensus_queue.cc:237] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } }
I20260923 21:00:51.855150 5054 catalog_manager.cc:5495] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e567a2310df4216aa950f1220ab4bda (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1e567a2310df4216aa950f1220ab4bda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e567a2310df4216aa950f1220ab4bda" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 } health_report { overall_health: UNKNOWN } } }
I20260923 21:00:51.881494 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.878530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.0.97.3:40141)
0923 21:00:51.878769 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.878778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.878894 (+ 116us) server_negotiation.cc:407] Connection header received
0923 21:00:51.879175 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.879180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.879270 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.879397 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.880699 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.880703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.880706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.880723 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.880757 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.880770 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.880923 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.881150 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.881218 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 21:00:51.890684 5328 raft_consensus.cc:1261] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 1 FOLLOWER]: Refusing update from remote peer 1e567a2310df4216aa950f1220ab4bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:51.890699 5197 raft_consensus.cc:1261] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 1 FOLLOWER]: Refusing update from remote peer 1e567a2310df4216aa950f1220ab4bda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:00:51.891779 5531 consensus_queue.cc:1035] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b4e2da53d943b18d66903c0b9c7e76" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:51.892165 5527 consensus_queue.cc:1035] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d43d0739a748ff8441d7fd4969fdf7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:00:51.923659 5054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:51.923969 5054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:00:51.927860 5054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d34ecc18b624e39bcd3568f78624485: Sending DeleteTablet for 3 replicas of tablet 8d5bdf5c07104ac3a0d435e5e353d8f5
I20260923 21:00:51.929175 5177 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:51 UTC) from {username='slave'} at 127.0.0.1:40920
I20260923 21:00:51.929400 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:51 UTC) from {username='slave'} at 127.0.0.1:59954
I20260923 21:00:51.930733 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:00:51 UTC) from {username='slave'} at 127.0.0.1:40758
I20260923 21:00:51.931749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5115
I20260923 21:00:51.940618 5545 tablet_replica.cc:330] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: stopping tablet replica
I20260923 21:00:51.940997 5544 tablet_replica.cc:330] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: stopping tablet replica
I20260923 21:00:51.941545 5545 raft_consensus.cc:2229] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:51.941957 5544 raft_consensus.cc:2229] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:00:51.942361 5545 raft_consensus.cc:2258] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:51.942545 5544 raft_consensus.cc:2258] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:51.945178 5544 ts_tablet_manager.cc:1905] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:51.945194 5545 ts_tablet_manager.cc:1905] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:51.953933 5544 ts_tablet_manager.cc:1918] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:51.954190 5544 log.cc:1196] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/8d5bdf5c07104ac3a0d435e5e353d8f5
I20260923 21:00:51.954725 5545 ts_tablet_manager.cc:1918] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:51.954872 5544 ts_tablet_manager.cc:1939] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 31d43d0739a748ff8441d7fd4969fdf7: Deleting consensus metadata
I20260923 21:00:51.955097 5545 log.cc:1196] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/8d5bdf5c07104ac3a0d435e5e353d8f5
I20260923 21:00:51.955801 5545 ts_tablet_manager.cc:1939] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P f8b4e2da53d943b18d66903c0b9c7e76: Deleting consensus metadata
I20260923 21:00:51.957134 5042 catalog_manager.cc:4841] TS 31d43d0739a748ff8441d7fd4969fdf7 (127.0.97.1:37477): tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 (table test-table [id=f78438752afa48839928bec7ed79cc26]) successfully deleted
I20260923 21:00:51.958149 5546 tablet_replica.cc:330] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: stopping tablet replica
I20260923 21:00:51.958164 5042 catalog_manager.cc:4841] TS f8b4e2da53d943b18d66903c0b9c7e76 (127.0.97.2:40583): tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 (table test-table [id=f78438752afa48839928bec7ed79cc26]) successfully deleted
I20260923 21:00:51.958740 5546 raft_consensus.cc:2229] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:00:51.959517 5546 raft_consensus.cc:2258] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:00:51.959725 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:51.933519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.0.97.1:37477)
0923 21:00:51.933720 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:51.933727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:51.956643 (+ 22916us) server_negotiation.cc:407] Connection header received
0923 21:00:51.957213 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:51.957219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:51.957328 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:51.957414 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:51.958874 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:51.958881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:51.958886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:51.958911 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:51.958967 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:51.958991 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:51.959155 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:51.959287 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 21:00:51.959359 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 21:00:51.960522 5237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:51.961874 5546 ts_tablet_manager.cc:1905] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:00:51.972998 5546 ts_tablet_manager.cc:1918] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:00:51.973284 5546 log.cc:1196] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/8d5bdf5c07104ac3a0d435e5e353d8f5
I20260923 21:00:51.974009 5546 ts_tablet_manager.cc:1939] T 8d5bdf5c07104ac3a0d435e5e353d8f5 P 1e567a2310df4216aa950f1220ab4bda: Deleting consensus metadata
I20260923 21:00:51.976598 5042 catalog_manager.cc:4841] TS 1e567a2310df4216aa950f1220ab4bda (127.0.97.3:40141): tablet 8d5bdf5c07104ac3a0d435e5e353d8f5 (table test-table [id=f78438752afa48839928bec7ed79cc26]) successfully deleted
I20260923 21:00:52.057099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5115
I20260923 21:00:52.083174 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5246
I20260923 21:00:52.086673 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:52.084135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.0.97.2:40583)
0923 21:00:52.084314 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:52.084318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:52.084340 (+ 22us) server_negotiation.cc:407] Connection header received
0923 21:00:52.084466 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:52.084470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:52.084564 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:52.084655 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:52.085812 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:52.085820 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:52.085826 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:52.085848 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:52.085885 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:52.085901 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:52.086026 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:52.086274 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 21:00:52.086353 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:00:52.087673 5368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:52.186888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5246
I20260923 21:00:52.214083 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5376
I20260923 21:00:52.218838 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:52.215120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.0.97.3:40141)
0923 21:00:52.215446 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:52.215453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:52.215480 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:00:52.215771 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:52.215777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:52.215852 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:52.216011 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:52.217414 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:52.217421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:52.217427 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:52.217451 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:52.217503 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:52.217522 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:52.217686 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:52.218372 (+ 686us) server_negotiation.cc:299] Negotiation successful
0923 21:00:52.218456 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260923 21:00:52.220155 5498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:52.343554 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5376
I20260923 21:00:52.362811 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5025
I20260923 21:00:52.364212 5086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:52.475886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5025
2026-09-23T21:00:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2905 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 21:00:52.519493 388 mini_oidc.cc:152] Starting JWKS server
I20260923 21:00:52.522238 388 webserver.cc:460] Webserver started at https://127.0.0.1:45085/ using document root <none> and password file <none>
I20260923 21:00:52.522676 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 21:00:52.523593 388 webserver.cc:460] Webserver started at http://127.0.0.1:39127/ using document root <none> and password file <none>
2026-09-23T21:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:52Z Disabled control of system clock
I20260923 21:00:52.554280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41847
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41847
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45085/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/testchainca.cert with env {}
W20260923 21:00:52.906663 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:52.906991 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:52.907151 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:52.917728 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:52.917904 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:52.917969 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:52.918048 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:52.918138 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 21:00:52.918228 5557 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 21:00:52.918303 5557 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45085/jwks.json
I20260923 21:00:52.929693 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45085/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:52.931341 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:52.932962 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:52.947332 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:52.947378 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:52.947420 5565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:52.948764 5557 server_base.cc:1034] running on GCE node
I20260923 21:00:52.949862 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:52.951995 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:52.953313 5557 hybrid_clock.cc:648] HybridClock initialized: now 1790197252953261 us; error 61 us; skew 500 ppm
I20260923 21:00:52.953810 5557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:52.956961 5557 webserver.cc:460] Webserver started at http://127.0.97.62:40065/ using document root <none> and password file <none>
I20260923 21:00:52.957840 5557 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:52.958007 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:52.958429 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:52.961668 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "5e31c427c73d40e3a5820c5dcfeabfd5"
format_stamp: "Formatted at 2026-09-23 21:00:52 on dist-test-slave-8xn5"
server_key: "2ee9d964095314ef25b5c69626342903"
server_key_iv: "0b264b35c6a3eab42058fc421e2ed3a4"
server_key_version: "encryptionkey@0"
I20260923 21:00:52.962685 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e31c427c73d40e3a5820c5dcfeabfd5"
format_stamp: "Formatted at 2026-09-23 21:00:52 on dist-test-slave-8xn5"
server_key: "2ee9d964095314ef25b5c69626342903"
server_key_iv: "0b264b35c6a3eab42058fc421e2ed3a4"
server_key_version: "encryptionkey@0"
I20260923 21:00:52.969264 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:00:52.974610 5571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:52.977010 5557 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 21:00:52.977383 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "5e31c427c73d40e3a5820c5dcfeabfd5"
format_stamp: "Formatted at 2026-09-23 21:00:52 on dist-test-slave-8xn5"
server_key: "2ee9d964095314ef25b5c69626342903"
server_key_iv: "0b264b35c6a3eab42058fc421e2ed3a4"
server_key_version: "encryptionkey@0"
I20260923 21:00:52.977727 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:53.022217 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:53.023265 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:53.023638 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:53.046044 5557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41847
I20260923 21:00:53.046165 5622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41847 every 8 connection(s)
I20260923 21:00:53.047741 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:53.052683 5623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:53.057705 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5557
I20260923 21:00:53.058125 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:53.058898 388 external_mini_cluster.cc:1428] Setting key 04c3f34e23793ec50f9fecbc0c1e0329
I20260923 21:00:53.065177 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5: Bootstrap starting.
I20260923 21:00:53.069494 5623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:53.070966 5623 log.cc:824] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:53.074606 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5: No bootstrap required, opened a new log
I20260923 21:00:53.083607 5623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41847 } }
I20260923 21:00:53.084101 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:53.061261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.0.97.62:41847)
0923 21:00:53.062628 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:53.062650 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:53.062705 (+ 55us) server_negotiation.cc:407] Connection header received
0923 21:00:53.064566 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:53.064601 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:53.065155 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:53.065638 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:53.066431 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.067872 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.070005 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.070972 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.071177 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:53.080008 (+ 8831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:53.080025 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:53.080086 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:53.080990 (+ 904us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:53.081104 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:53.081134 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:53.081254 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:53.081693 (+ 439us) server_negotiation.cc:299] Negotiation successful
0923 21:00:53.082182 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":493,"threads_started":1}
I20260923 21:00:53.084106 5623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:53.084293 5623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e31c427c73d40e3a5820c5dcfeabfd5, State: Initialized, Role: FOLLOWER
I20260923 21:00:53.085011 5623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41847 } }
I20260923 21:00:53.085222 5623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:53.085314 5623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:53.085518 5623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:53.088400 5623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41847 } }
I20260923 21:00:53.088759 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:00:53.089098 5623 leader_election.cc:302] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e31c427c73d40e3a5820c5dcfeabfd5; no voters:
I20260923 21:00:53.089810 5623 leader_election.cc:288] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:53.089950 5628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:53.091594 5628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [term 1 LEADER]: Becoming Leader. State: Replica: 5e31c427c73d40e3a5820c5dcfeabfd5, State: Running, Role: LEADER
I20260923 21:00:53.092247 5628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41847 } }
I20260923 21:00:53.092664 5623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:53.096016 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41847 } } }
I20260923 21:00:53.096474 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:53.097012 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e31c427c73d40e3a5820c5dcfeabfd5. Latest consensus state: current_term: 1 leader_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e31c427c73d40e3a5820c5dcfeabfd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41847 } } }
I20260923 21:00:53.097909 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:53.099308 5636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:53.105120 5636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:53.115288 5636 catalog_manager.cc:1347] Generated new cluster ID: b94dace5ae7d4845baddff369ba4d492
I20260923 21:00:53.115434 5636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:53.135013 5636 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:53.136750 5636 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:53.153919 5636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e31c427c73d40e3a5820c5dcfeabfd5: Generated new TSK 0
I20260923 21:00:53.155009 5636 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 21:00:53.204730 5653 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 21:00:53.204928 5653 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 21:00:53.205756 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:53.173753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.0.97.62:41847)
0923 21:00:53.174114 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:53.174125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:53.174704 (+ 579us) server_negotiation.cc:407] Connection header received
0923 21:00:53.175109 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:53.175115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:53.175216 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:53.175393 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0923 21:00:53.176210 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.177406 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.179808 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.180280 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.180395 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:53.181162 (+ 767us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 21:00:53.205205 (+ 24043us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 21:00:53.205537 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260923 21:00:53.205925 5626 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 21:00:53.207968 5652 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:00:53.173695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41847 (local address 127.0.0.1:58892)
0923 21:00:53.174557 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:53.174581 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:53.174801 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:53.175516 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:53.175522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:53.175536 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 21:00:53.175996 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:53.176010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.177598 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.177604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:53.179582 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:53.179604 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.180437 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.180526 (+ 89us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:53.180727 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:53.180828 (+ 101us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 21:00:53.205544 (+ 24716us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 21:00:53.205932 (+ 388us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":392,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:41847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45085/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 21:00:53.210085 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5557
I20260923 21:00:53.211488 5618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:53.314517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5557
2026-09-23T21:00:53Z chronyd exiting
I20260923 21:00:53.366674 388 test_util.cc:175] -----------------------------------------------
I20260923 21:00:53.366869 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790197207951953-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (852 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 21:00:53.370934 388 mini_oidc.cc:152] Starting JWKS server
I20260923 21:00:53.372911 388 webserver.cc:460] Webserver started at https://127.0.0.1:44223/ using document root <none> and password file <none>
I20260923 21:00:53.373173 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 21:00:53.373870 388 webserver.cc:460] Webserver started at http://127.0.0.1:37087/ using document root <none> and password file <none>
2026-09-23T21:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:53Z Disabled control of system clock
I20260923 21:00:53.403076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41671
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44223/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/testchainca.cert with env {}
W20260923 21:00:53.757110 5660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:53.757511 5660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:53.757705 5660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:53.765549 5660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:53.765726 5660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:53.765813 5660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:53.765925 5660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:53.766013 5660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 21:00:53.766103 5660 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 21:00:53.766180 5660 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44223/jwks.json
I20260923 21:00:53.777268 5660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44223/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:53.779069 5660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:53.780521 5660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:53.794711 5665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:53.794840 5666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:53.795064 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:53.796357 5660 server_base.cc:1034] running on GCE node
I20260923 21:00:53.797272 5660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:53.799126 5660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:53.800390 5660 hybrid_clock.cc:648] HybridClock initialized: now 1790197253800344 us; error 46 us; skew 500 ppm
I20260923 21:00:53.800861 5660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:53.803791 5660 webserver.cc:460] Webserver started at http://127.0.97.62:34375/ using document root <none> and password file <none>
I20260923 21:00:53.804642 5660 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:53.804790 5660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:53.805177 5660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:53.808527 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "48f7724b40f34aa88a2a7185c48930fb"
format_stamp: "Formatted at 2026-09-23 21:00:53 on dist-test-slave-8xn5"
server_key: "2e22e5eacd04af1651ba7facf6245ca3"
server_key_iv: "8780f11718615fa5557f6b88f11e268b"
server_key_version: "encryptionkey@0"
I20260923 21:00:53.809504 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "48f7724b40f34aa88a2a7185c48930fb"
format_stamp: "Formatted at 2026-09-23 21:00:53 on dist-test-slave-8xn5"
server_key: "2e22e5eacd04af1651ba7facf6245ca3"
server_key_iv: "8780f11718615fa5557f6b88f11e268b"
server_key_version: "encryptionkey@0"
I20260923 21:00:53.815980 5660 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 21:00:53.820461 5674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:53.822248 5660 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 21:00:53.822522 5660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "48f7724b40f34aa88a2a7185c48930fb"
format_stamp: "Formatted at 2026-09-23 21:00:53 on dist-test-slave-8xn5"
server_key: "2e22e5eacd04af1651ba7facf6245ca3"
server_key_iv: "8780f11718615fa5557f6b88f11e268b"
server_key_version: "encryptionkey@0"
I20260923 21:00:53.822779 5660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:53.852843 5660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:53.853821 5660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:53.854138 5660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:53.875373 5660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41671
I20260923 21:00:53.875391 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41671 every 8 connection(s)
I20260923 21:00:53.876932 5660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:53.881387 5726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:53.886061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5660
I20260923 21:00:53.886305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:53.887030 388 external_mini_cluster.cc:1428] Setting key 0408cfc0e72e853c7b905586dc0e7689
I20260923 21:00:53.893745 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb: Bootstrap starting.
I20260923 21:00:53.897428 5726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:53.898808 5726 log.cc:824] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:53.902666 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb: No bootstrap required, opened a new log
I20260923 21:00:53.902711 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:53.889181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.0.97.62:41671)
0923 21:00:53.890391 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:53.890418 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:53.890470 (+ 52us) server_negotiation.cc:407] Connection header received
0923 21:00:53.892365 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:53.892392 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:53.892983 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:53.893519 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:53.894178 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.895689 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.897153 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.898056 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.898286 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:53.899680 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:53.899696 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:53.899725 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:53.900648 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:53.900738 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:53.900763 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:53.900872 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:53.901205 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 21:00:53.901595 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":369,"threads_started":1}
I20260923 21:00:53.907586 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:00:53.912753 5726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f7724b40f34aa88a2a7185c48930fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41671 } }
I20260923 21:00:53.913221 5726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:53.913347 5726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f7724b40f34aa88a2a7185c48930fb, State: Initialized, Role: FOLLOWER
I20260923 21:00:53.913995 5726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f7724b40f34aa88a2a7185c48930fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41671 } }
I20260923 21:00:53.914268 5726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:53.914439 5726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:53.914629 5726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:53.917078 5726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f7724b40f34aa88a2a7185c48930fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41671 } }
I20260923 21:00:53.917680 5726 leader_election.cc:302] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f7724b40f34aa88a2a7185c48930fb; no voters:
I20260923 21:00:53.918442 5726 leader_election.cc:288] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:53.918670 5731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:53.919620 5731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [term 1 LEADER]: Becoming Leader. State: Replica: 48f7724b40f34aa88a2a7185c48930fb, State: Running, Role: LEADER
I20260923 21:00:53.920192 5731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f7724b40f34aa88a2a7185c48930fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41671 } }
I20260923 21:00:53.920744 5726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:53.923636 5733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f7724b40f34aa88a2a7185c48930fb. Latest consensus state: current_term: 1 leader_uuid: "48f7724b40f34aa88a2a7185c48930fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f7724b40f34aa88a2a7185c48930fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41671 } } }
I20260923 21:00:53.924050 5733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:53.924572 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f7724b40f34aa88a2a7185c48930fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f7724b40f34aa88a2a7185c48930fb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41671 } } }
I20260923 21:00:53.924911 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:53.926676 5737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:53.932231 5737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:53.941876 5737 catalog_manager.cc:1347] Generated new cluster ID: 7bf26b83d09e419185fc4e724213ceab
I20260923 21:00:53.942032 5737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:53.950845 5737 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:53.953131 5737 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:53.964653 5737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f7724b40f34aa88a2a7185c48930fb: Generated new TSK 0
I20260923 21:00:53.965950 5737 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 21:00:54.003350 5756 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 21:00:54.003546 5756 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 21:00:54.004817 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:53.982118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.0.97.62:41671)
0923 21:00:53.982385 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:53.982394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:53.983057 (+ 663us) server_negotiation.cc:407] Connection header received
0923 21:00:53.983319 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:53.983324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:53.983448 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:53.983676 (+ 228us) server_negotiation.cc:226] Negotiated authn=JWT
0923 21:00:53.984270 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.985456 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.987063 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.987834 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.987982 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:53.988612 (+ 630us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 21:00:54.004076 (+ 15464us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 21:00:54.004507 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34840: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":89}
W20260923 21:00:54.004962 5729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34840: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 21:00:54.005224 5755 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:00:53.982083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41671 (local address 127.0.0.1:34840)
0923 21:00:53.982901 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:53.982927 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:53.983091 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:53.983745 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:53.983750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:53.983763 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 21:00:53.984050 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:53.984059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.985658 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.985663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:53.986901 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:53.986915 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:53.988000 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:53.988054 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:53.988257 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:53.988307 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 21:00:54.004402 (+ 16095us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 21:00:54.004633 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44223/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":485,"threads_started":1}
I20260923 21:00:54.007011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5660
I20260923 21:00:54.008482 5721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:54.103124 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5660
2026-09-23T21:00:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (790 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 21:00:54.162029 388 mini_oidc.cc:152] Starting JWKS server
I20260923 21:00:54.164114 388 webserver.cc:460] Webserver started at https://127.0.0.1:45143/ using document root <none> and password file <none>
I20260923 21:00:54.164422 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 21:00:54.165146 388 webserver.cc:460] Webserver started at http://127.0.0.1:43791/ using document root <none> and password file <none>
2026-09-23T21:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:54Z Disabled control of system clock
I20260923 21:00:54.196712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34493
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34493
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45143/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 21:00:54.554677 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:54.555004 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:54.555154 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:54.563057 5763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:54.563185 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:54.563248 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:54.563339 5763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:54.563436 5763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 21:00:54.563541 5763 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 21:00:54.563596 5763 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45143/jwks.json
I20260923 21:00:54.573519 5763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45143/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:54.575001 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:54.576884 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:54.590421 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:54.590390 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:54.590461 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:54.591312 5763 server_base.cc:1034] running on GCE node
I20260923 21:00:54.592176 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:54.593766 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:54.595036 5763 hybrid_clock.cc:648] HybridClock initialized: now 1790197254594973 us; error 64 us; skew 500 ppm
I20260923 21:00:54.595434 5763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:54.598399 5763 webserver.cc:460] Webserver started at http://127.0.97.62:38341/ using document root <none> and password file <none>
I20260923 21:00:54.599165 5763 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:54.599298 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:54.599663 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:54.602836 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "a625f55508a24a51a50d503cc2906e25"
format_stamp: "Formatted at 2026-09-23 21:00:54 on dist-test-slave-8xn5"
server_key: "4284075f36fad93e169a26e7a5e4f888"
server_key_iv: "d62ecb6603e324c41959b5581100834a"
server_key_version: "encryptionkey@0"
I20260923 21:00:54.603703 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "a625f55508a24a51a50d503cc2906e25"
format_stamp: "Formatted at 2026-09-23 21:00:54 on dist-test-slave-8xn5"
server_key: "4284075f36fad93e169a26e7a5e4f888"
server_key_iv: "d62ecb6603e324c41959b5581100834a"
server_key_version: "encryptionkey@0"
I20260923 21:00:54.609367 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 21:00:54.613716 5777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:54.615366 5763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 21:00:54.615583 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "a625f55508a24a51a50d503cc2906e25"
format_stamp: "Formatted at 2026-09-23 21:00:54 on dist-test-slave-8xn5"
server_key: "4284075f36fad93e169a26e7a5e4f888"
server_key_iv: "d62ecb6603e324c41959b5581100834a"
server_key_version: "encryptionkey@0"
I20260923 21:00:54.615824 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:54.633142 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:54.634079 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:54.634447 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:54.654484 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34493 every 8 connection(s)
I20260923 21:00:54.654501 5763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34493
I20260923 21:00:54.656032 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:54.659747 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5763
I20260923 21:00:54.659976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:54.660209 5829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:54.660676 388 external_mini_cluster.cc:1428] Setting key 68ae2d751cd0f3143cb00ccd8fced2a2
I20260923 21:00:54.675925 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25: Bootstrap starting.
I20260923 21:00:54.677140 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:54.662715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.97.62:34493)
0923 21:00:54.663918 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:54.663940 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:54.663997 (+ 57us) server_negotiation.cc:407] Connection header received
0923 21:00:54.665848 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:54.665879 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:54.666582 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:54.667060 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:54.667643 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:54.669708 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:54.671490 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:54.672635 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:54.672850 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:54.674107 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:54.674117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:54.674138 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:54.674941 (+ 803us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:54.675042 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:54.675074 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:54.675193 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:54.675426 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 21:00:54.675770 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":342,"threads_started":1}
I20260923 21:00:54.679642 5829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:54.680830 5829 log.cc:824] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:54.681123 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:00:54.684540 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25: No bootstrap required, opened a new log
I20260923 21:00:54.692729 5829 raft_consensus.cc:348] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a625f55508a24a51a50d503cc2906e25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34493 } }
I20260923 21:00:54.693261 5829 raft_consensus.cc:374] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:54.693409 5829 raft_consensus.cc:729] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a625f55508a24a51a50d503cc2906e25, State: Initialized, Role: FOLLOWER
I20260923 21:00:54.694108 5829 consensus_queue.cc:260] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a625f55508a24a51a50d503cc2906e25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34493 } }
I20260923 21:00:54.694372 5829 raft_consensus.cc:388] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:54.694537 5829 raft_consensus.cc:482] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:54.694707 5829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:54.696923 5829 raft_consensus.cc:504] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a625f55508a24a51a50d503cc2906e25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34493 } }
I20260923 21:00:54.697539 5829 leader_election.cc:302] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a625f55508a24a51a50d503cc2906e25; no voters:
I20260923 21:00:54.698232 5829 leader_election.cc:288] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:54.698401 5834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:54.699172 5834 raft_consensus.cc:686] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [term 1 LEADER]: Becoming Leader. State: Replica: a625f55508a24a51a50d503cc2906e25, State: Running, Role: LEADER
I20260923 21:00:54.699862 5834 consensus_queue.cc:237] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a625f55508a24a51a50d503cc2906e25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34493 } }
I20260923 21:00:54.700906 5829 sys_catalog.cc:564] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:54.703020 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a625f55508a24a51a50d503cc2906e25. Latest consensus state: current_term: 1 leader_uuid: "a625f55508a24a51a50d503cc2906e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a625f55508a24a51a50d503cc2906e25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34493 } } }
I20260923 21:00:54.703094 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a625f55508a24a51a50d503cc2906e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a625f55508a24a51a50d503cc2906e25" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34493 } } }
I20260923 21:00:54.703433 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:54.703523 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:54.705724 5841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:54.711077 5841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:54.720573 5841 catalog_manager.cc:1347] Generated new cluster ID: 28e9f552706e4437ad74b24d20f5dd3f
I20260923 21:00:54.720780 5841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:54.743505 5841 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:54.744932 5841 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:54.752547 5841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a625f55508a24a51a50d503cc2906e25: Generated new TSK 0
I20260923 21:00:54.753509 5841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:00:54.781368 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:54.771638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33402 (local address 127.0.97.62:34493)
0923 21:00:54.771885 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:54.771895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:54.772453 (+ 558us) server_negotiation.cc:407] Connection header received
0923 21:00:54.772714 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:54.772718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:54.772796 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:54.772950 (+ 154us) server_negotiation.cc:226] Negotiated authn=JWT
0923 21:00:54.773537 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:54.774443 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:54.776729 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:54.777226 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:54.777430 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:54.778054 (+ 624us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 21:00:54.780691 (+ 2637us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 21:00:54.781058 (+ 367us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 21:00:54.781679 5832 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 21:00:54.784219 5858 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:00:54.771620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34493 (local address 127.0.0.1:33402)
0923 21:00:54.772311 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:54.772335 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:54.772528 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:54.772992 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:54.772997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:54.773011 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 21:00:54.773367 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:54.773381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:54.774607 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:54.774614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:54.776517 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:54.776536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:54.777373 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:54.777459 (+ 86us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:54.777663 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:54.777713 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 21:00:54.780963 (+ 3250us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 21:00:54.783713 (+ 2750us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34493: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45143/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":2276,"thread_start_us":355,"threads_started":1}
I20260923 21:00:54.786291 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5763
I20260923 21:00:54.788034 5824 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 21:00:54.800397 5859 webserver.cc:533] Webserver: sslize failed:
E20260923 21:00:54.800591 5859 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 21:00:54.886569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5763
2026-09-23T21:00:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (777 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T21:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:54Z Disabled control of system clock
I20260923 21:00:54.968670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44727
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44727
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/testchainca.cert with env {}
W20260923 21:00:55.341673 5864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:55.342008 5864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:55.342097 5864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:55.349622 5864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:55.349747 5864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:55.349786 5864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:55.349835 5864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:55.349879 5864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 21:00:55.349941 5864 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 21:00:55.349982 5864 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 21:00:55.360193 5864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:55.361562 5864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:55.362967 5864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:55.376827 5869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:55.376895 5870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:55.376819 5872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:55.380223 5864 server_base.cc:1034] running on GCE node
I20260923 21:00:55.381141 5864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:55.382899 5864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:55.384155 5864 hybrid_clock.cc:648] HybridClock initialized: now 1790197255384080 us; error 72 us; skew 500 ppm
I20260923 21:00:55.384555 5864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:00:55.387312 5864 webserver.cc:460] Webserver started at http://127.0.97.62:46261/ using document root <none> and password file <none>
I20260923 21:00:55.388106 5864 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:55.388257 5864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:55.388597 5864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:55.391855 5864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "687e8372efdb4645a1b6dfc5927f7f00"
format_stamp: "Formatted at 2026-09-23 21:00:55 on dist-test-slave-8xn5"
server_key: "3ebcad3341fb63eea74d176096c10141"
server_key_iv: "3b7a4e67c65d03eccb5c4a191810f698"
server_key_version: "encryptionkey@0"
I20260923 21:00:55.392743 5864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "687e8372efdb4645a1b6dfc5927f7f00"
format_stamp: "Formatted at 2026-09-23 21:00:55 on dist-test-slave-8xn5"
server_key: "3ebcad3341fb63eea74d176096c10141"
server_key_iv: "3b7a4e67c65d03eccb5c4a191810f698"
server_key_version: "encryptionkey@0"
I20260923 21:00:55.398504 5864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 21:00:55.402518 5878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:55.404258 5864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:00:55.404620 5864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "687e8372efdb4645a1b6dfc5927f7f00"
format_stamp: "Formatted at 2026-09-23 21:00:55 on dist-test-slave-8xn5"
server_key: "3ebcad3341fb63eea74d176096c10141"
server_key_iv: "3b7a4e67c65d03eccb5c4a191810f698"
server_key_version: "encryptionkey@0"
I20260923 21:00:55.404878 5864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:55.440256 5864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:55.441236 5864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:55.441593 5864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:55.461436 5864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44727
I20260923 21:00:55.461473 5929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44727 every 8 connection(s)
I20260923 21:00:55.462945 5864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:55.467211 5930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:55.472332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5864
I20260923 21:00:55.472498 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:55.473253 388 external_mini_cluster.cc:1428] Setting key 149687196bd149c48d673d4abceb2b6b
I20260923 21:00:55.480561 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00: Bootstrap starting.
I20260923 21:00:55.484920 5930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00: Neither blocks nor log segments found. Creating new log.
I20260923 21:00:55.486558 5930 log.cc:824] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:55.490438 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:55.475510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.0.97.62:44727)
0923 21:00:55.476833 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:55.476845 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:55.476892 (+ 47us) server_negotiation.cc:407] Connection header received
0923 21:00:55.478836 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:55.478862 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:55.479394 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:55.479762 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:55.480535 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:55.482088 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:55.484311 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:55.485241 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:55.485418 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:55.487089 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:55.487097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:55.487114 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:00:55.488183 (+ 1069us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:55.488306 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:00:55.488343 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:55.488490 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:55.488851 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 21:00:55.489276 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":563,"threads_started":1}
I20260923 21:00:55.490891 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00: No bootstrap required, opened a new log
I20260923 21:00:55.494977 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:00:55.501875 5930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687e8372efdb4645a1b6dfc5927f7f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44727 } }
I20260923 21:00:55.502525 5930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:55.502671 5930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687e8372efdb4645a1b6dfc5927f7f00, State: Initialized, Role: FOLLOWER
I20260923 21:00:55.503435 5930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687e8372efdb4645a1b6dfc5927f7f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44727 } }
I20260923 21:00:55.503677 5930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:55.503778 5930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:55.503938 5930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:55.506466 5930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687e8372efdb4645a1b6dfc5927f7f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44727 } }
I20260923 21:00:55.507107 5930 leader_election.cc:302] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 687e8372efdb4645a1b6dfc5927f7f00; no voters:
I20260923 21:00:55.507786 5930 leader_election.cc:288] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:55.508060 5935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:55.508860 5935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [term 1 LEADER]: Becoming Leader. State: Replica: 687e8372efdb4645a1b6dfc5927f7f00, State: Running, Role: LEADER
I20260923 21:00:55.509533 5935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687e8372efdb4645a1b6dfc5927f7f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44727 } }
I20260923 21:00:55.510458 5930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:55.512696 5937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687e8372efdb4645a1b6dfc5927f7f00. Latest consensus state: current_term: 1 leader_uuid: "687e8372efdb4645a1b6dfc5927f7f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687e8372efdb4645a1b6dfc5927f7f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44727 } } }
I20260923 21:00:55.513049 5937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:55.514267 5940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:55.516101 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "687e8372efdb4645a1b6dfc5927f7f00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687e8372efdb4645a1b6dfc5927f7f00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44727 } } }
I20260923 21:00:55.516628 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:55.524080 5940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:55.537497 5940 catalog_manager.cc:1347] Generated new cluster ID: b407bf99e3cb4d5c8007241415195bfc
I20260923 21:00:55.537709 5940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:55.552408 5940 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:55.554268 5940 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:55.573180 5940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687e8372efdb4645a1b6dfc5927f7f00: Generated new TSK 0
I20260923 21:00:55.574239 5940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:00:55.663424 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:55.591212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.0.97.62:44727)
0923 21:00:55.591443 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:55.591452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:55.591872 (+ 420us) server_negotiation.cc:407] Connection header received
0923 21:00:55.592159 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:55.592164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:55.592243 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:55.592435 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0923 21:00:55.592977 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:55.593804 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:55.596046 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:55.596584 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:55.596736 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:55.597303 (+ 567us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 21:00:55.662604 (+ 65301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 21:00:55.663089 (+ 485us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57716: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 21:00:55.663604 5933 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57716: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 21:00:55.663727 5959 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:00:55.591062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44727 (local address 127.0.0.1:57716)
0923 21:00:55.591750 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:55.591768 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:55.591965 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:55.592450 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:55.592455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:55.592468 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 21:00:55.592790 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:55.592800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:55.593979 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:55.593993 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:55.595869 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:55.595885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:55.596737 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:55.596801 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:55.596970 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:55.597009 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 21:00:55.662932 (+ 65923us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 21:00:55.663243 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44727: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":384,"threads_started":1}
I20260923 21:00:55.665387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5864
I20260923 21:00:55.666857 5925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:00:55.769870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5864
2026-09-23T21:00:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (865 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (447 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (450 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:00:56 dist-test-slave-8xn5 krb5kdc[5974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:00:56 dist-test-slave-8xn5 krb5kdc[5974](info): set up 2 sockets
Sep 23 21:00:56 dist-test-slave-8xn5 krb5kdc[5974](info): commencing operation
krb5kdc: starting...
W20260923 21:00:57.839341 388 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:00:57 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:00:58.080039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32839
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:58.435828 5987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:58.436154 5987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:58.436342 5987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:58.445546 5987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:00:58.445685 5987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:58.445816 5987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:58.445909 5987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:00:58.445982 5987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:00:58.456561 5987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:58.458048 5987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:58.459686 5987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:58.473410 5992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:58.473366 5993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:58.473315 5995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:58.475009 5987 server_base.cc:1034] running on GCE node
I20260923 21:00:58.475957 5987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:58.477607 5987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:58.478850 5987 hybrid_clock.cc:648] HybridClock initialized: now 1790197258478782 us; error 75 us; skew 500 ppm
I20260923 21:00:58.479236 5987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:58 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:58.487085 5987 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:00:58.489202 5987 webserver.cc:460] Webserver started at http://127.0.97.62:37679/ using document root <none> and password file <none>
I20260923 21:00:58.490042 5987 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:58.490173 5987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:58.490559 5987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:58.493798 5987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "c1bd88bb268441598fa1fb4332f4e8f0"
format_stamp: "Formatted at 2026-09-23 21:00:58 on dist-test-slave-8xn5"
server_key: "fa416b6b95af85fe2b25001482b3064b"
server_key_iv: "a9dfef91fc5d734dc28fc20a38f60c0e"
server_key_version: "encryptionkey@0"
I20260923 21:00:58.494751 5987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "c1bd88bb268441598fa1fb4332f4e8f0"
format_stamp: "Formatted at 2026-09-23 21:00:58 on dist-test-slave-8xn5"
server_key: "fa416b6b95af85fe2b25001482b3064b"
server_key_iv: "a9dfef91fc5d734dc28fc20a38f60c0e"
server_key_version: "encryptionkey@0"
I20260923 21:00:58.501255 5987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:00:58.505470 6002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:58.507325 5987 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:58.507561 5987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "c1bd88bb268441598fa1fb4332f4e8f0"
format_stamp: "Formatted at 2026-09-23 21:00:58 on dist-test-slave-8xn5"
server_key: "fa416b6b95af85fe2b25001482b3064b"
server_key_iv: "a9dfef91fc5d734dc28fc20a38f60c0e"
server_key_version: "encryptionkey@0"
I20260923 21:00:58.507771 5987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:58.527920 5987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:58.533241 5987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:58.533604 5987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:58.556401 5987 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32839
I20260923 21:00:58.556430 6053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32839 every 8 connection(s)
I20260923 21:00:58.558012 5987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:00:58.562700 6054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:00:58.564678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 5987
I20260923 21:00:58.564917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:00:58.565805 388 external_mini_cluster.cc:1428] Setting key d06b4141bf85afd4010f2a3ea8992c61
I20260923 21:00:58.578202 6054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0: Bootstrap starting.
I20260923 21:00:58.583055 6054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0: Neither blocks nor log segments found. Creating new log.
Sep 23 21:00:58 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:58.584578 6054 log.cc:824] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0: Log is configured to *not* fsync() on all Append() calls
I20260923 21:00:58.588383 6054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0: No bootstrap required, opened a new log
I20260923 21:00:58.593484 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:58.568449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34832 (local address 127.0.97.62:32839)
0923 21:00:58.569499 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:58.569517 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:58.569567 (+ 50us) server_negotiation.cc:407] Connection header received
0923 21:00:58.571708 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:58.571753 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:58.572222 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:58.572628 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:58.574887 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:58.576467 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:58.578588 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:58.579567 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:58.579788 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:58.585224 (+ 5436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:58.585244 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:58.585270 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:58.585344 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:58.588853 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:58.589696 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:58.589705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:58.589713 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:58.589896 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:58.590376 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:58.590383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:58.590386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:58.590998 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:58.591347 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:58.591822 (+ 475us) server_negotiation.cc:299] Negotiation successful
0923 21:00:58.592316 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":537,"threads_started":1}
I20260923 21:00:58.597648 6054 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260923 21:00:58.598115 6054 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:00:58.598203 6054 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bd88bb268441598fa1fb4332f4e8f0, State: Initialized, Role: FOLLOWER
I20260923 21:00:58.598860 6054 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260923 21:00:58.599123 6054 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:00:58.599313 6054 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:00:58.599504 6054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:00:58.602092 6054 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260923 21:00:58.602741 6054 leader_election.cc:302] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1bd88bb268441598fa1fb4332f4e8f0; no voters:
I20260923 21:00:58.603433 6054 leader_election.cc:288] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:00:58.603641 6059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:00:58.604368 6059 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [term 1 LEADER]: Becoming Leader. State: Replica: c1bd88bb268441598fa1fb4332f4e8f0, State: Running, Role: LEADER
I20260923 21:00:58.604998 6059 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } }
I20260923 21:00:58.605990 6054 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:00:58.608381 6061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1bd88bb268441598fa1fb4332f4e8f0. Latest consensus state: current_term: 1 leader_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } } }
I20260923 21:00:58.608314 6060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bd88bb268441598fa1fb4332f4e8f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32839 } } }
I20260923 21:00:58.608819 6061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:58.608819 6060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0 [sys.catalog]: This master's current role is: LEADER
I20260923 21:00:58.609752 6065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:00:58.619989 6065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:00:58.629587 6065 catalog_manager.cc:1347] Generated new cluster ID: 324c5e9d58cb4f90a4b921a863743f60
I20260923 21:00:58.629732 6065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:00:58.656106 6065 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:00:58.658217 6065 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:00:58.674604 6065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1bd88bb268441598fa1fb4332f4e8f0: Generated new TSK 0
I20260923 21:00:58.675884 6065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:00:58.739558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32839
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:59.110050 6080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:59.110430 6080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:59.110540 6080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:59.118321 6080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:59.118531 6080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:59.118650 6080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:00:59.129016 6080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:59.130409 6080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:59.131754 6080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:59.146961 6085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:59.146993 6086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:59.147038 6088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:59.147562 6080 server_base.cc:1034] running on GCE node
I20260923 21:00:59.148463 6080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:59.149593 6080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:59.150960 6080 hybrid_clock.cc:648] HybridClock initialized: now 1790197259150886 us; error 63 us; skew 500 ppm
I20260923 21:00:59.151360 6080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:59 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:59.155356 6080 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:00:59.157229 6080 webserver.cc:460] Webserver started at http://127.0.97.1:35049/ using document root <none> and password file <none>
I20260923 21:00:59.158070 6080 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:59.158252 6080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:59.158641 6080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:59.162249 6080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "67f7571d65f74857b7bc1e44892711ff"
format_stamp: "Formatted at 2026-09-23 21:00:59 on dist-test-slave-8xn5"
server_key: "3d6ad139fb3fd2ce6d29ae7e6ed79ab5"
server_key_iv: "3cbc2d5bda6ba125d01819c66ddc7a8b"
server_key_version: "encryptionkey@0"
I20260923 21:00:59.163348 6080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "67f7571d65f74857b7bc1e44892711ff"
format_stamp: "Formatted at 2026-09-23 21:00:59 on dist-test-slave-8xn5"
server_key: "3d6ad139fb3fd2ce6d29ae7e6ed79ab5"
server_key_iv: "3cbc2d5bda6ba125d01819c66ddc7a8b"
server_key_version: "encryptionkey@0"
I20260923 21:00:59.169896 6080 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 21:00:59.174146 6095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:59.176016 6080 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:00:59.176241 6080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "67f7571d65f74857b7bc1e44892711ff"
format_stamp: "Formatted at 2026-09-23 21:00:59 on dist-test-slave-8xn5"
server_key: "3d6ad139fb3fd2ce6d29ae7e6ed79ab5"
server_key_iv: "3cbc2d5bda6ba125d01819c66ddc7a8b"
server_key_version: "encryptionkey@0"
I20260923 21:00:59.176477 6080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:59.207616 6080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:59.213163 6080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:59.213495 6080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:59.214593 6080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:59.216005 6080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:59.216112 6080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:59.216269 6080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:59.216342 6080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:59.252442 6080 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46621
I20260923 21:00:59.252481 6207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46621 every 8 connection(s)
I20260923 21:00:59.253891 6080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:00:59.264391 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 6080
I20260923 21:00:59.264670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:00:59.265340 388 external_mini_cluster.cc:1428] Setting key 1740fb13d115f8e44703845444fdb09f
Sep 23 21:00:59 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:59.279515 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:59.256860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51979 (local address 127.0.97.62:32839)
0923 21:00:59.257338 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:59.257345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:59.258732 (+ 1387us) server_negotiation.cc:407] Connection header received
0923 21:00:59.260664 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:59.260669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:59.260744 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:59.260855 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:59.262818 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.263580 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.265953 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.266600 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.266714 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:59.271710 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:59.271721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:59.271730 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:59.271794 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:59.275532 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:59.276760 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:59.276765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:59.276768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:59.276920 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:59.277524 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:59.277528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:59.277530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:59.277835 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:59.278040 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:59.279054 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0923 21:00:59.279248 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260923 21:00:59.281220 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:59.257457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32839 (local address 127.0.97.1:51979)
0923 21:00:59.258439 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:59.258487 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:59.260339 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:59.261151 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:59.261161 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:59.261853 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:59.262602 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:59.262624 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.263867 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.263872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:59.265753 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:59.265771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.266881 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.266889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:59.267255 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:59.268377 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:59.268418 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:59.271362 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:59.275757 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:59.275771 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:59.275796 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:59.276563 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:59.277051 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:59.277059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:59.277066 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:59.277372 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:59.277981 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:59.277994 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:59.278534 (+ 540us) client_negotiation.cc:769] Sending connection context
0923 21:00:59.278901 (+ 367us) client_negotiation.cc:240] Negotiation successful
0923 21:00:59.279550 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":385,"threads_started":1}
I20260923 21:00:59.283285 6208 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32839
I20260923 21:00:59.283692 6208 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:59.284719 6208 heartbeater.cc:511] Master 127.0.97.62:32839 requested a full tablet report, sending...
I20260923 21:00:59.287381 6019 ts_manager.cc:194] Registered new tserver with Master: 67f7571d65f74857b7bc1e44892711ff (127.0.97.1:46621)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:00:59.289037 6019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:00:59.319212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32839
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:00:59.681748 6214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:00:59.682127 6214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:00:59.682308 6214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:00:59.690845 6214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:00:59.691045 6214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:00:59.691223 6214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:00:59.702216 6214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:00:59.703727 6214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:00:59.705303 6214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:00:59.722007 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:59.721953 6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:00:59.722054 6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:00:59.723030 6214 server_base.cc:1034] running on GCE node
I20260923 21:00:59.723913 6214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:00:59.724994 6214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:00:59.726265 6214 hybrid_clock.cc:648] HybridClock initialized: now 1790197259726173 us; error 90 us; skew 500 ppm
I20260923 21:00:59.726686 6214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:00:59 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:00:59.730293 6214 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:00:59.732082 6214 webserver.cc:460] Webserver started at http://127.0.97.2:34165/ using document root <none> and password file <none>
I20260923 21:00:59.732802 6214 fs_manager.cc:359] Metadata directory not provided
I20260923 21:00:59.732911 6214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:00:59.733176 6214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:00:59.736317 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "fcc057e08a9241e2ad9e04c586ec61bf"
format_stamp: "Formatted at 2026-09-23 21:00:59 on dist-test-slave-8xn5"
server_key: "4394f2374e092cfaf47258c401978c07"
server_key_iv: "1cf70234f1f4834bd9025bd0a5a6877b"
server_key_version: "encryptionkey@0"
I20260923 21:00:59.737139 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fcc057e08a9241e2ad9e04c586ec61bf"
format_stamp: "Formatted at 2026-09-23 21:00:59 on dist-test-slave-8xn5"
server_key: "4394f2374e092cfaf47258c401978c07"
server_key_iv: "1cf70234f1f4834bd9025bd0a5a6877b"
server_key_version: "encryptionkey@0"
I20260923 21:00:59.743309 6214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 21:00:59.747452 6229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:59.749090 6214 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:00:59.749300 6214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "fcc057e08a9241e2ad9e04c586ec61bf"
format_stamp: "Formatted at 2026-09-23 21:00:59 on dist-test-slave-8xn5"
server_key: "4394f2374e092cfaf47258c401978c07"
server_key_iv: "1cf70234f1f4834bd9025bd0a5a6877b"
server_key_version: "encryptionkey@0"
I20260923 21:00:59.749590 6214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:00:59.785146 6214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:00:59.790508 6214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:00:59.790827 6214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:00:59.791882 6214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:00:59.793447 6214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:00:59.793524 6214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:59.793715 6214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:00:59.793767 6214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:00:59.830648 6214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34125
I20260923 21:00:59.830842 6341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34125 every 8 connection(s)
I20260923 21:00:59.832041 6214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:00:59.842141 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 6214
I20260923 21:00:59.842422 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:00:59.843019 388 external_mini_cluster.cc:1428] Setting key 69bed81d642306d0de5872ee2bbda62d
Sep 23 21:00:59 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:00:59.856765 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:59.835043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43483 (local address 127.0.97.62:32839)
0923 21:00:59.835567 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0923 21:00:59.835572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:00:59.836581 (+ 1009us) server_negotiation.cc:407] Connection header received
0923 21:00:59.838513 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:00:59.838519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:00:59.838618 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:00:59.838789 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:00:59.840444 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.841171 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.843245 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.843883 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.844044 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:59.848167 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:00:59.848179 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:00:59.848187 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:00:59.848257 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:00:59.852445 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:59.853734 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:59.853740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:59.853745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:59.853929 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:00:59.854646 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:00:59.854650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:00:59.854652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:00:59.855057 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:00:59.855191 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 21:00:59.856256 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0923 21:00:59.856480 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":264,"threads_started":1}
I20260923 21:00:59.858528 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:00:59.835602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32839 (local address 127.0.97.2:43483)
0923 21:00:59.836365 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0923 21:00:59.836421 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 21:00:59.838138 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:00:59.839066 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:00:59.839075 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:00:59.839630 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:00:59.840233 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:59.840253 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.841345 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.841349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:59.843060 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:00:59.843075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:00:59.844144 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:00:59.844150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:00:59.844453 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:00:59.845302 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:00:59.845344 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:00:59.847841 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:00:59.852669 (+ 4828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:59.852682 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:59.852707 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:59.853489 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:59.854092 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:00:59.854100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:00:59.854107 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:00:59.854461 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:00:59.855200 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:00:59.855213 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:00:59.855768 (+ 555us) client_negotiation.cc:769] Sending connection context
0923 21:00:59.856122 (+ 354us) client_negotiation.cc:240] Negotiation successful
0923 21:00:59.856807 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":312,"threads_started":1}
I20260923 21:00:59.860256 6342 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32839
I20260923 21:00:59.860661 6342 heartbeater.cc:464] Registering TS with master...
I20260923 21:00:59.861505 6342 heartbeater.cc:511] Master 127.0.97.62:32839 requested a full tablet report, sending...
I20260923 21:00:59.863448 6019 ts_manager.cc:194] Registered new tserver with Master: fcc057e08a9241e2ad9e04c586ec61bf (127.0.97.2:34125)
I20260923 21:00:59.864871 6019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43483
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:00:59.898221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32839
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:00.266894 6348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:00.267236 6348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:00.267407 6348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:00.276041 6348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:00.276182 6348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:00.276389 6348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:00.289085 6348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:00.290735 6348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:00.292464 6348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 21:01:00.293160 6208 heartbeater.cc:503] Master 127.0.97.62:32839 was elected leader, sending a full tablet report...
W20260923 21:01:00.310375 6356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:00.310448 6354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:00.310390 6353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:00.311359 6348 server_base.cc:1034] running on GCE node
I20260923 21:01:00.312073 6348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:00.313216 6348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:00.314452 6348 hybrid_clock.cc:648] HybridClock initialized: now 1790197260314377 us; error 71 us; skew 500 ppm
I20260923 21:01:00.314790 6348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.318199 6348 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:01:00.320210 6348 webserver.cc:460] Webserver started at http://127.0.97.3:37111/ using document root <none> and password file <none>
I20260923 21:01:00.320998 6348 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:00.321161 6348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:00.321520 6348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:00.324831 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "146c1ae47ccf458ab3c9f6569e05adb6"
format_stamp: "Formatted at 2026-09-23 21:01:00 on dist-test-slave-8xn5"
server_key: "70578c72a976f340abf7aa0f9f71391d"
server_key_iv: "075cd360d26e7ced151771261b0ec2e8"
server_key_version: "encryptionkey@0"
I20260923 21:01:00.325731 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "146c1ae47ccf458ab3c9f6569e05adb6"
format_stamp: "Formatted at 2026-09-23 21:01:00 on dist-test-slave-8xn5"
server_key: "70578c72a976f340abf7aa0f9f71391d"
server_key_iv: "075cd360d26e7ced151771261b0ec2e8"
server_key_version: "encryptionkey@0"
I20260923 21:01:00.331558 6348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 21:01:00.335675 6363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:00.337359 6348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260923 21:01:00.337587 6348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "146c1ae47ccf458ab3c9f6569e05adb6"
format_stamp: "Formatted at 2026-09-23 21:01:00 on dist-test-slave-8xn5"
server_key: "70578c72a976f340abf7aa0f9f71391d"
server_key_iv: "075cd360d26e7ced151771261b0ec2e8"
server_key_version: "encryptionkey@0"
I20260923 21:01:00.337831 6348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:00.364238 6348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:00.369074 6348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:00.369414 6348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:00.370517 6348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:00.371929 6348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:00.372035 6348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:00.372186 6348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:00.372259 6348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:00.409315 6348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46671
I20260923 21:01:00.409420 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46671 every 8 connection(s)
I20260923 21:01:00.410951 6348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:00.411885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 6348
I20260923 21:01:00.412091 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:01:00.412679 388 external_mini_cluster.cc:1428] Setting key 5a7da658835cd96a81dd8025b55b1337
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:00.434453 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.414840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33279 (local address 127.0.97.62:32839)
0923 21:01:00.415384 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.415390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.417387 (+ 1997us) server_negotiation.cc:407] Connection header received
0923 21:01:00.419351 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.419357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.419460 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.419578 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.421174 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.421950 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.423672 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.424083 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.424225 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.427928 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:00.427933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:00.427937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:00.427973 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:00.431125 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.431952 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.431961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.431964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.432083 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.432548 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.432553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.432558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.432844 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:00.433004 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:00.433796 (+ 792us) server_negotiation.cc:299] Negotiation successful
0923 21:01:00.434028 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":310,"threads_started":1}
I20260923 21:01:00.435463 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.415730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32839 (local address 127.0.97.3:33279)
0923 21:01:00.417232 (+ 1502us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:00.417281 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:00.419060 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:00.419829 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:00.419837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:00.420396 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:00.420992 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:00.421009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.422120 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.422124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:00.423508 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:00.423522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.424337 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.424345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:00.424595 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.425324 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:00.425348 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:00.427655 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:00.431296 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:00.431307 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:00.431327 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:00.431812 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:00.432208 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:00.432212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:00.432216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:00.432425 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:00.432986 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:00.432995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:00.433377 (+ 382us) client_negotiation.cc:769] Sending connection context
0923 21:01:00.433675 (+ 298us) client_negotiation.cc:240] Negotiation successful
0923 21:01:00.434268 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":475,"threads_started":1}
I20260923 21:01:00.437170 6476 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32839
I20260923 21:01:00.437517 6476 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:00.438292 6476 heartbeater.cc:511] Master 127.0.97.62:32839 requested a full tablet report, sending...
I20260923 21:01:00.439941 6019 ts_manager.cc:194] Registered new tserver with Master: 146c1ae47ccf458ab3c9f6569e05adb6 (127.0.97.3:46671)
I20260923 21:01:00.440948 6019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33279
I20260923 21:01:00.442865 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/test-admin.keytab.
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.473544 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 21:01:00.479277 6484 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:00.476509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.97.1:46621)
0923 21:01:00.477246 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.477253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.477279 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:01:00.477339 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.477345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.477582 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.477714 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.478741 (+ 1027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57304: BlockingRecv error: recv got EOF from 127.0.0.1:57304 (error 108)
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":457,"threads_started":1}
W20260923 21:01:00.479887 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.482004 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.484095 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:01:00.502316 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.487072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.0.97.1:46621)
0923 21:01:00.487258 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.487263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.487288 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:01:00.487393 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.487397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.487462 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.487550 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.488700 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.489851 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.491351 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.492012 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.492126 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.496257 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:00.496272 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:00.496289 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:00.496344 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:00.499416 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.500189 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.500197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.500200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.500329 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.500750 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.500753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.500756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.501150 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:00.501406 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:00.501794 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 21:01:00.502037 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 21:01:00.509105 6484 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:00.507289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.0.97.1:46621)
0923 21:01:00.507512 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.507519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.507702 (+ 183us) server_negotiation.cc:407] Connection header received
0923 21:01:00.507920 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.507932 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.507992 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.508093 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.508851 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57332: BlockingRecv error: recv got EOF from 127.0.0.1:57332 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260923 21:01:00.509316 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.511047 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.512782 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:01:00.528772 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.514471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.0.97.1:46621)
0923 21:01:00.514675 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.514684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.515170 (+ 486us) server_negotiation.cc:407] Connection header received
0923 21:01:00.515406 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.515410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.515474 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.515583 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.516789 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.517550 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.518961 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.519408 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.519541 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.523456 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:00.523462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:00.523465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:00.523506 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:00.526017 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.526873 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.526883 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.526888 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.527023 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.527436 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.527440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.527442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.527708 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:00.527979 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:00.528277 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 21:01:00.528490 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 21:01:00.533833 6484 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:00.531732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.0.97.1:46621)
0923 21:01:00.531899 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.531904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.532389 (+ 485us) server_negotiation.cc:407] Connection header received
0923 21:01:00.532520 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.532523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.532596 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.532678 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.533548 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57340: BlockingRecv error: recv got EOF from 127.0.0.1:57340 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 21:01:00.534089 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790197207951953-388-0/krb5kdc/krb5cc' of type FILE
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.535830 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:00.537218 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:01:00.554432 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.538736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.0.97.1:46621)
0923 21:01:00.539072 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.539079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.539258 (+ 179us) server_negotiation.cc:407] Connection header received
0923 21:01:00.539464 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.539470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.539543 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.539651 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.540662 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.541399 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.542872 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.543380 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.543516 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.547623 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:00.547630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:00.547633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:00.547756 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:00.550731 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.551701 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.551705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.551708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.551878 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.552543 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.552548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.552550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.552999 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:00.553310 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:00.553839 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 21:01:00.554038 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260923 21:01:00.556582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 6080
I20260923 21:01:00.573053 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.557532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.0.97.1:46621)
0923 21:01:00.557732 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.557738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.558311 (+ 573us) server_negotiation.cc:407] Connection header received
0923 21:01:00.558568 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.558574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.558659 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.558774 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.560058 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.561242 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.563637 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.564139 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.564261 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.567185 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:00.567193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:00.567198 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:00.567243 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:00.570109 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.571011 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.571016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.571019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.571147 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.571557 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.571560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.571562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.571852 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:00.572043 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:00.572404 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 21:01:00.572584 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 21:01:00.573640 6203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:00.675355 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 6080
I20260923 21:01:00.692472 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 6214
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:01:00.710229 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.693560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.0.97.2:34125)
0923 21:01:00.694092 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.694100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.694125 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:01:00.694190 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.694197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.694487 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.694625 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.695755 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.696921 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.698268 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.699151 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.699281 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.703463 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:00.703518 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:00.703541 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:00.703602 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:00.707101 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.707926 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.707935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.707937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.708076 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.708473 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.708476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.708478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.708948 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:00.709180 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:00.709505 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 21:01:00.709905 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260923 21:01:00.711702 6337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:00.806200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 6214
I20260923 21:01:00.823006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 6348
Sep 23 21:01:00 dist-test-slave-8xn5 krb5kdc[5974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:01:00.840231 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:00.824052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.0.97.3:46671)
0923 21:01:00.824699 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:00.824706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:00.824735 (+ 29us) server_negotiation.cc:407] Connection header received
0923 21:01:00.824793 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:00.824799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:00.825029 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:00.825167 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:00.826269 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.827398 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.828766 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:00.829352 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:00.829457 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:00.833791 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:00.833807 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:00.833824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:00.833906 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:00.836936 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.837780 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.837789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.837791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.837940 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:00.838510 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:00.838514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:00.838517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:00.838947 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:00.839283 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:00.839612 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 21:01:00.839901 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":331,"threads_started":1}
I20260923 21:01:00.841660 6471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:00.928526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 6348
I20260923 21:01:00.945171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 5987
I20260923 21:01:00.946830 6049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:01.052112 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 5987
2026-09-23T21:01:01Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4393 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:01:01 dist-test-slave-8xn5 krb5kdc[6504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:01:01 dist-test-slave-8xn5 krb5kdc[6504](info): set up 2 sockets
Sep 23 21:01:01 dist-test-slave-8xn5 krb5kdc[6504](info): commencing operation
krb5kdc: starting...
W20260923 21:01:02.247395 388 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:01:02 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260923 21:01:02.479164 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39963
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:02.819512 6517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:02.819911 6517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:02.820068 6517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:02.827740 6517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:02.827912 6517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:02.828020 6517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:02.828131 6517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:02.828210 6517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:02.838557 6517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:02.840009 6517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:02.841415 6517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:02.858989 6522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:02.858948 6523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:02.859025 6525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:02.860086 6517 server_base.cc:1034] running on GCE node
I20260923 21:01:02.860987 6517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:02.862864 6517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:02.864147 6517 hybrid_clock.cc:648] HybridClock initialized: now 1790197262864089 us; error 47 us; skew 500 ppm
I20260923 21:01:02.864686 6517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:02 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:02.868805 6517 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260923 21:01:02.871006 6517 webserver.cc:460] Webserver started at http://127.0.97.62:35185/ using document root <none> and password file <none>
I20260923 21:01:02.871905 6517 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:02.872097 6517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:02.872478 6517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:02.876289 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "709239c2e0d0464882bf6056e5b9c0b3"
format_stamp: "Formatted at 2026-09-23 21:01:02 on dist-test-slave-8xn5"
server_key: "3a714f5fe4fa9da2a6641b7a70b220a4"
server_key_iv: "de8c5fa9e84230694f902691ec7a5042"
server_key_version: "encryptionkey@0"
I20260923 21:01:02.877301 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "709239c2e0d0464882bf6056e5b9c0b3"
format_stamp: "Formatted at 2026-09-23 21:01:02 on dist-test-slave-8xn5"
server_key: "3a714f5fe4fa9da2a6641b7a70b220a4"
server_key_iv: "de8c5fa9e84230694f902691ec7a5042"
server_key_version: "encryptionkey@0"
I20260923 21:01:02.884068 6517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 21:01:02.888605 6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:02.890465 6517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:01:02.890694 6517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "709239c2e0d0464882bf6056e5b9c0b3"
format_stamp: "Formatted at 2026-09-23 21:01:02 on dist-test-slave-8xn5"
server_key: "3a714f5fe4fa9da2a6641b7a70b220a4"
server_key_iv: "de8c5fa9e84230694f902691ec7a5042"
server_key_version: "encryptionkey@0"
I20260923 21:01:02.890947 6517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:02.911839 6517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:02.917037 6517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:02.917388 6517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:02.944024 6517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39963
I20260923 21:01:02.944056 6583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39963 every 8 connection(s)
I20260923 21:01:02.945649 6517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:01:02.949869 6584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:02.952137 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 6517
I20260923 21:01:02.952366 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:01:02.953115 388 external_mini_cluster.cc:1428] Setting key 105b6575ced0b7888c4e31505a980a8e
I20260923 21:01:02.964207 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: Bootstrap starting.
I20260923 21:01:02.968226 6584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:02.969395 6584 log.cc:824] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:01:02 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:02.972924 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: No bootstrap required, opened a new log
I20260923 21:01:02.979889 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:02.955474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.0.97.62:39963)
0923 21:01:02.956401 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:02.956420 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:02.956475 (+ 55us) server_negotiation.cc:407] Connection header received
0923 21:01:02.958908 (+ 2433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:02.958945 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:02.959550 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:02.960114 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:02.961581 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:02.963630 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:02.965603 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:02.966860 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:02.967074 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:02.971584 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:02.971593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:02.971612 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:02.971651 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:02.975316 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:02.976157 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:02.976165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:02.976173 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:02.976363 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:02.976775 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:02.976780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:02.976784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:02.977386 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:02.977764 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:02.978230 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 21:01:02.978726 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":397,"threads_started":1}
I20260923 21:01:02.981295 6584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } }
I20260923 21:01:02.981820 6584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:02.981946 6584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709239c2e0d0464882bf6056e5b9c0b3, State: Initialized, Role: FOLLOWER
I20260923 21:01:02.982679 6584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } }
I20260923 21:01:02.982980 6584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:01:02.983127 6584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:01:02.983330 6584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:02.985556 6584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } }
I20260923 21:01:02.986137 6584 leader_election.cc:302] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 709239c2e0d0464882bf6056e5b9c0b3; no voters:
I20260923 21:01:02.986837 6584 leader_election.cc:288] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:01:02.987022 6589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:02.988507 6589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 1 LEADER]: Becoming Leader. State: Replica: 709239c2e0d0464882bf6056e5b9c0b3, State: Running, Role: LEADER
I20260923 21:01:02.989185 6589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } }
I20260923 21:01:02.989326 6584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:02.993008 6591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709239c2e0d0464882bf6056e5b9c0b3. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } }
I20260923 21:01:02.992980 6590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } }
I20260923 21:01:02.993474 6591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:02.993475 6590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:02.995942 6597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:03.002079 6597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:03.013353 6597 catalog_manager.cc:1347] Generated new cluster ID: 6b5f8a05b56b494e9540ffb2aa37be8d
I20260923 21:01:03.013494 6597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:03.025377 6597 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:01:03.027012 6597 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:03.035481 6597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: Generated new TSK 0
I20260923 21:01:03.036697 6597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260923 21:01:03.107712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39963
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:03.467341 6610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:03.467662 6610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:03.467773 6610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:03.475720 6610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:03.475869 6610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:03.476053 6610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:03.486924 6610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:03.488389 6610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:03.489985 6610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:03.505894 6618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:03.505985 6615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:03.506383 6616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:03.506999 6610 server_base.cc:1034] running on GCE node
I20260923 21:01:03.507684 6610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:03.508801 6610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:03.510026 6610 hybrid_clock.cc:648] HybridClock initialized: now 1790197263509967 us; error 53 us; skew 500 ppm
I20260923 21:01:03.510635 6610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:03 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:03.521200 6610 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 21:01:03.523326 6610 webserver.cc:460] Webserver started at http://127.0.97.1:35359/ using document root <none> and password file <none>
I20260923 21:01:03.524201 6610 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:03.524358 6610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:03.524735 6610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:03.528254 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "cdba6177147a47e693ad2d152127f63a"
format_stamp: "Formatted at 2026-09-23 21:01:03 on dist-test-slave-8xn5"
server_key: "4569643d76b8c4dbdbc0dca4117b0d0b"
server_key_iv: "2d79b1171246e07e4e205799952b97fa"
server_key_version: "encryptionkey@0"
I20260923 21:01:03.529201 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cdba6177147a47e693ad2d152127f63a"
format_stamp: "Formatted at 2026-09-23 21:01:03 on dist-test-slave-8xn5"
server_key: "4569643d76b8c4dbdbc0dca4117b0d0b"
server_key_iv: "2d79b1171246e07e4e205799952b97fa"
server_key_version: "encryptionkey@0"
I20260923 21:01:03.536471 6610 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260923 21:01:03.540913 6625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:03.542879 6610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:01:03.543092 6610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "cdba6177147a47e693ad2d152127f63a"
format_stamp: "Formatted at 2026-09-23 21:01:03 on dist-test-slave-8xn5"
server_key: "4569643d76b8c4dbdbc0dca4117b0d0b"
server_key_iv: "2d79b1171246e07e4e205799952b97fa"
server_key_version: "encryptionkey@0"
I20260923 21:01:03.543335 6610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:03.571475 6610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:03.576383 6610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:03.576733 6610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:03.577865 6610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:03.579346 6610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:03.579437 6610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:03.579574 6610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:03.579644 6610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:03.616345 6610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38547
I20260923 21:01:03.616400 6737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38547 every 8 connection(s)
I20260923 21:01:03.617669 6610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:03.622081 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 6610
I20260923 21:01:03.622385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:01:03.623220 388 external_mini_cluster.cc:1428] Setting key 6f434e175c92eef1f1eaf68e3b512721
Sep 23 21:01:03 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197263, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:03.643883 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:03.620453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41985 (local address 127.0.97.62:39963)
0923 21:01:03.620967 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:03.620973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:03.622387 (+ 1414us) server_negotiation.cc:407] Connection header received
0923 21:01:03.624906 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:03.624911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:03.624998 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:03.625141 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:03.627476 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:03.628320 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:03.631028 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:03.631433 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:03.631566 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:03.635713 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:03.635723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:03.635731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:03.635787 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:03.639722 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:03.640634 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:03.640643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:03.640650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:03.640823 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:03.641368 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:03.641376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:03.641382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:03.641857 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:03.642050 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:03.642991 (+ 941us) server_negotiation.cc:299] Negotiation successful
0923 21:01:03.643347 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":272,"threads_started":1}
I20260923 21:01:03.644577 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:03.621165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:41985)
0923 21:01:03.622178 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:03.622234 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:03.624590 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:03.625440 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:03.625452 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:03.626278 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:03.627222 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:03.627253 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:03.628496 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:03.628504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:03.630815 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:03.630836 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:03.631716 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:03.631726 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:03.632093 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:03.633001 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:03.633026 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:03.635396 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:03.639929 (+ 4533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:03.639938 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:03.639954 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:03.640453 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:03.640971 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:03.640976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:03.640978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:03.641211 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:03.642046 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:03.642053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:03.642497 (+ 444us) client_negotiation.cc:769] Sending connection context
0923 21:01:03.642778 (+ 281us) client_negotiation.cc:240] Negotiation successful
0923 21:01:03.643260 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":437,"threads_started":1}
I20260923 21:01:03.646564 6738 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39963
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:01:03.646989 6738 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:03.647904 6738 heartbeater.cc:511] Master 127.0.97.62:39963 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260923 21:01:03.650738 6549 ts_manager.cc:194] Registered new tserver with Master: cdba6177147a47e693ad2d152127f63a (127.0.97.1:38547)
I20260923 21:01:03.652287 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260923 21:01:03.678936 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39963
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:04.033123 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:04.033448 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:04.033545 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:04.041241 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:04.041388 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:04.041507 6744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:04.052146 6744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:04.053501 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:04.055192 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:04.071206 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:04.071293 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:04.071224 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:04.076772 6744 server_base.cc:1034] running on GCE node
I20260923 21:01:04.077675 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:04.079087 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:04.080350 6744 hybrid_clock.cc:648] HybridClock initialized: now 1790197264080275 us; error 72 us; skew 500 ppm
I20260923 21:01:04.080865 6744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:04 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197264, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:04.084589 6744 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 21:01:04.086570 6744 webserver.cc:460] Webserver started at http://127.0.97.2:34669/ using document root <none> and password file <none>
I20260923 21:01:04.087386 6744 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:04.087529 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:04.087894 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:04.091444 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "5e5abb06795f44babcdb3f3f4040a426"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "09c019cb27c825452ac48deecc0e592f"
server_key_iv: "6aa19610a5011d72b018ef242d9fafc7"
server_key_version: "encryptionkey@0"
I20260923 21:01:04.092410 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5e5abb06795f44babcdb3f3f4040a426"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "09c019cb27c825452ac48deecc0e592f"
server_key_iv: "6aa19610a5011d72b018ef242d9fafc7"
server_key_version: "encryptionkey@0"
I20260923 21:01:04.099131 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 21:01:04.103459 6759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:04.105397 6744 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:01:04.105618 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "5e5abb06795f44babcdb3f3f4040a426"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "09c019cb27c825452ac48deecc0e592f"
server_key_iv: "6aa19610a5011d72b018ef242d9fafc7"
server_key_version: "encryptionkey@0"
I20260923 21:01:04.105877 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:04.125130 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:04.130049 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:04.130312 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:04.131372 6744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:04.133025 6744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:04.133106 6744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:04.133199 6744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:04.133244 6744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:04.168222 6744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41743
I20260923 21:01:04.168262 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41743 every 8 connection(s)
I20260923 21:01:04.169605 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:04.173774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 6744
I20260923 21:01:04.173971 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:01:04.174620 388 external_mini_cluster.cc:1428] Setting key 23ea33e10de20f6f00eea7c4e6247305
Sep 23 21:01:04 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197264, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:04.194298 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:04.172441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42771 (local address 127.0.97.62:39963)
0923 21:01:04.172892 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:04.172896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:04.174419 (+ 1523us) server_negotiation.cc:407] Connection header received
0923 21:01:04.176463 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:04.176468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:04.176557 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:04.176661 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:04.178650 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.179833 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.181685 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.182483 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.182636 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:04.187228 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:04.187235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:04.187238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:04.187279 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:04.190410 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:04.191381 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:04.191387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:04.191392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:04.191542 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:04.192049 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:04.192053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:04.192055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:04.192535 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:04.192843 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:04.193637 (+ 794us) server_negotiation.cc:299] Negotiation successful
0923 21:01:04.193906 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":238,"threads_started":1}
I20260923 21:01:04.195568 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:04.173038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:42771)
0923 21:01:04.173852 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:04.173946 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:04.176052 (+ 2106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:04.177155 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:04.177165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:04.177726 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:04.178434 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:04.178457 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.180026 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.180030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:04.181500 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:04.181513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.182711 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.182716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:04.182953 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:04.183741 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:04.183844 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:04.186869 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:04.190610 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:04.190628 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:04.190646 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:04.191230 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:04.191677 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:04.191682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:04.191686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:04.191910 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:04.192707 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:04.192717 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:04.193164 (+ 447us) client_negotiation.cc:769] Sending connection context
0923 21:01:04.193522 (+ 358us) client_negotiation.cc:240] Negotiation successful
0923 21:01:04.194122 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":348,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:01:04.197556 6872 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39963
I20260923 21:01:04.197968 6872 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:04.199060 6872 heartbeater.cc:511] Master 127.0.97.62:39963 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260923 21:01:04.201133 6549 ts_manager.cc:194] Registered new tserver with Master: 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:04.202127 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 21:01:04.230957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39963
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:04.582247 6878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:04.582677 6878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:04.582835 6878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:04.590749 6878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:04.590883 6878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:04.591058 6878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:04.602582 6878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:04.604178 6878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:04.605684 6878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:04.621405 6884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:04.621551 6886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:04.621956 6883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:04.622725 6878 server_base.cc:1034] running on GCE node
I20260923 21:01:04.623523 6878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:04.624574 6878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:04.625844 6878 hybrid_clock.cc:648] HybridClock initialized: now 1790197264625769 us; error 68 us; skew 500 ppm
I20260923 21:01:04.626216 6878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:04 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197264, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:04.629724 6878 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 21:01:04.631479 6878 webserver.cc:460] Webserver started at http://127.0.97.3:37765/ using document root <none> and password file <none>
I20260923 21:01:04.632275 6878 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:04.632434 6878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:04.632771 6878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:04.635936 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "ddf0c67eda0742cda4369580b7ba83fa"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "dfcd89d27bc6f661a25de71d2a40f7e0"
server_key_iv: "e7333091c5e498ac76af8b129756a6c6"
server_key_version: "encryptionkey@0"
I20260923 21:01:04.636793 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ddf0c67eda0742cda4369580b7ba83fa"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "dfcd89d27bc6f661a25de71d2a40f7e0"
server_key_iv: "e7333091c5e498ac76af8b129756a6c6"
server_key_version: "encryptionkey@0"
I20260923 21:01:04.642730 6878 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 21:01:04.647101 6893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:04.649036 6878 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:01:04.649268 6878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "ddf0c67eda0742cda4369580b7ba83fa"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "dfcd89d27bc6f661a25de71d2a40f7e0"
server_key_iv: "e7333091c5e498ac76af8b129756a6c6"
server_key_version: "encryptionkey@0"
I20260923 21:01:04.649505 6878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:04.656852 6738 heartbeater.cc:503] Master 127.0.97.62:39963 was elected leader, sending a full tablet report...
I20260923 21:01:04.665757 6878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:04.670921 6878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:04.671247 6878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:04.672405 6878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:04.674124 6878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:04.674209 6878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:04.674422 6878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:04.674482 6878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:04.714046 6878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34747
I20260923 21:01:04.714085 7005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34747 every 8 connection(s)
I20260923 21:01:04.715571 6878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:04.725366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 6878
I20260923 21:01:04.725576 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:01:04.726078 388 external_mini_cluster.cc:1428] Setting key f5e7a3f851ecdc4b8877cd37006addca
Sep 23 21:01:04 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197264, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:04.738850 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:04.718464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34181 (local address 127.0.97.62:39963)
0923 21:01:04.718998 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:04.719004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:04.720149 (+ 1145us) server_negotiation.cc:407] Connection header received
0923 21:01:04.722289 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:04.722295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:04.722447 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:04.722583 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:04.724338 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.725236 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.727381 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.728080 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.728265 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:04.732561 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:04.732572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:04.732576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:04.732614 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:04.735552 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:04.736397 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:04.736402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:04.736404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:04.736523 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:04.737015 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:04.737019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:04.737021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:04.737281 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:04.737387 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:04.738221 (+ 834us) server_negotiation.cc:299] Negotiation successful
0923 21:01:04.738486 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
I20260923 21:01:04.740172 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:04.719077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:34181)
0923 21:01:04.719940 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:04.719984 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:04.721946 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:04.722879 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:04.722889 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:04.723491 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:04.724137 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:04.724157 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.725419 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.725424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:04.727216 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:04.727235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:04.728435 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:04.728443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:04.728749 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:04.729575 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:04.729607 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:04.732299 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:04.735753 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:04.735762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:04.735778 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:04.736260 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:04.736671 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:04.736675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:04.736678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:04.736897 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:04.737410 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:04.737418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:04.737830 (+ 412us) client_negotiation.cc:769] Sending connection context
0923 21:01:04.738105 (+ 275us) client_negotiation.cc:240] Negotiation successful
0923 21:01:04.738689 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":328,"threads_started":1}
I20260923 21:01:04.742075 7006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39963
I20260923 21:01:04.742522 7006 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:04.743412 7006 heartbeater.cc:511] Master 127.0.97.62:39963 requested a full tablet report, sending...
I20260923 21:01:04.744884 6549 ts_manager.cc:194] Registered new tserver with Master: ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747)
I20260923 21:01:04.745725 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34181
I20260923 21:01:04.756451 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260923 21:01:04.812187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33323
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39963,127.0.97.61:33323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:05.164000 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:05.164378 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:05.164546 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:05.174974 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:05.175148 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:05.175220 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:05.175287 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:05.175333 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:05.185840 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39963,127.0.97.61:33323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
W20260923 21:01:05.186244 7012 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:39963,127.0.97.61:33323'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 21:01:05.187457 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:05.188846 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:05.203420 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:05.203410 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:05.203419 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:05.204870 7012 server_base.cc:1034] running on GCE node
I20260923 21:01:05.205384 6872 heartbeater.cc:503] Master 127.0.97.62:39963 was elected leader, sending a full tablet report...
I20260923 21:01:05.205969 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:05.207896 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:05.209335 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790197265209147 us; error 47 us; skew 500 ppm
I20260923 21:01:05.209873 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:05 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:05.213619 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260923 21:01:05.215610 7012 webserver.cc:460] Webserver started at http://127.0.97.61:43835/ using document root <none> and password file <none>
I20260923 21:01:05.216394 7012 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:05.216495 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:05.216912 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:05.220528 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/instance:
uuid: "f90374f37df945d1805e836834117852"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "0ead0027c67d787ee103029e8c999c16"
server_key_iv: "e6dee8e1c2fca0cb97b6752063c73ab9"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.221423 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal/instance:
uuid: "f90374f37df945d1805e836834117852"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "0ead0027c67d787ee103029e8c999c16"
server_key_iv: "e6dee8e1c2fca0cb97b6752063c73ab9"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.227833 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 21:01:05.232309 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:05.234629 7012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20260923 21:01:05.234915 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
uuid: "f90374f37df945d1805e836834117852"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "0ead0027c67d787ee103029e8c999c16"
server_key_iv: "e6dee8e1c2fca0cb97b6752063c73ab9"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.235221 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:05.253015 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:05.258453 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:05.258769 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:05.279948 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33323
I20260923 21:01:05.279954 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33323 every 8 connection(s)
I20260923 21:01:05.281563 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
I20260923 21:01:05.285816 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:05.286461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 7012
I20260923 21:01:05.296381 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:01:05 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 21:01:05 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:05.315474 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.288901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.0.97.61:33323)
0923 21:01:05.289954 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.289968 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.290016 (+ 48us) server_negotiation.cc:407] Connection header received
0923 21:01:05.292794 (+ 2778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.292826 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.293353 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.293791 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:05.295885 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.297747 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.300148 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.301380 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.301566 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.306556 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:05.306566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:05.306584 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:05.306640 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:05.310776 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.311905 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.311912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.311921 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.312074 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.312482 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.312503 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.312505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.313053 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:05.313396 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.313830 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.314280 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":493,"threads_started":1}
I20260923 21:01:05.317135 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.299700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.0.97.62:39963)
0923 21:01:05.300321 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.300327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.300819 (+ 492us) server_negotiation.cc:407] Connection header received
0923 21:01:05.301053 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.301057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.301131 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.301269 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:05.302846 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.303675 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.305467 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.306156 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.306304 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.311126 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:05.311133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:05.311137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:05.311187 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:05.314065 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.314929 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.314935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.314939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.315077 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.315471 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.315476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.315480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.315934 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:05.316108 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.316561 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.316789 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":273,"threads_started":1}
I20260923 21:01:05.317240 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.299773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48020)
0923 21:01:05.300667 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:05.300724 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:05.300913 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:05.301335 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:05.301343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:05.302259 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:05.302654 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.302670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.303829 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.303836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.305293 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.305305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.306491 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.306498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.306736 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.307562 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:05.307584 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:05.310877 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:05.314275 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.314290 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.314299 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.314800 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.315203 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.315214 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.315216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.315350 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.316088 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:05.316095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:05.316262 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 21:01:05.316458 (+ 196us) client_negotiation.cc:240] Negotiation successful
0923 21:01:05.316784 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":505,"threads_started":1}
I20260923 21:01:05.319005 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:01:05 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 21:01:05.332700 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.319786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.0.1:53918)
0923 21:01:05.319952 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:05.319967 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:05.320047 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:05.320414 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:05.320417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:05.320771 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:05.320902 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.320909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.322010 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.322014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.322993 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.323004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.323762 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.323765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.323886 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.324569 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:05.324599 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:05.327006 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:05.330144 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.330149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.330154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.330663 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.331036 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.331041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.331044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.331172 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.331697 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:05.331702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:05.331850 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 21:01:05.331982 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 21:01:05.332287 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 21:01:05.332708 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.319971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.0.97.61:33323)
0923 21:01:05.320172 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.320176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.320191 (+ 15us) server_negotiation.cc:407] Connection header received
0923 21:01:05.320238 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.320241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.320294 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.320372 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:05.321031 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.321861 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.323156 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.323544 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.323652 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.327259 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:05.327264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:05.327267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:05.327296 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:05.329972 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.330791 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.330797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.330800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.330918 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.331284 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.331289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.331292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.331580 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:05.331732 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.332063 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.332416 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":61}
I20260923 21:01:05.337816 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Bootstrap starting.
I20260923 21:01:05.341410 7079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:05.342621 7079 log.cc:824] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:05.345551 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: No bootstrap required, opened a new log
I20260923 21:01:05.353482 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } }
I20260923 21:01:05.354007 7079 raft_consensus.cc:374] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:05.354127 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f90374f37df945d1805e836834117852, State: Initialized, Role: FOLLOWER
I20260923 21:01:05.354962 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } }
I20260923 21:01:05.355926 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } } }
I20260923 21:01:05.356278 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:01:05.357075 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 21:01:05.367187 7097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 21:01:05.367377 7097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 21:01:05.370400 7012 master_runner.cc:428] Detected that this master f90374f37df945d1805e836834117852 is joining an existing cluster
I20260923 21:01:05.370514 7012 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:33323
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260923 21:01:05.399351 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.380021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.0.97.62:39963)
0923 21:01:05.380232 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.380238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.381850 (+ 1612us) server_negotiation.cc:407] Connection header received
0923 21:01:05.382108 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.382113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.382232 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.382308 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:05.384147 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.385280 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.387667 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.388439 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.388552 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.391900 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:05.391906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:05.391911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:05.391955 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:05.395546 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.396609 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.396618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.396625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.396803 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.397299 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.397303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.397307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.397711 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:05.397872 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.398571 (+ 699us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.398886 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 21:01:05.399986 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.380115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.0.97.61:33323)
0923 21:01:05.380308 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.380316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.381762 (+ 1446us) server_negotiation.cc:407] Connection header received
0923 21:01:05.382014 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.382023 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.382092 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.382164 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:05.384121 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.385289 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.387701 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.388419 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.388533 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.391633 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:05.391641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:05.391645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:05.391679 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:05.395579 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.396722 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.396727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.396730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.396876 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.397356 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.397361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.397364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.398688 (+ 1324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:05.398846 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.399386 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.399627 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:01:05.400645 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.380635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48024)
0923 21:01:05.381723 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:05.381750 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:05.381948 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:05.382561 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:05.382566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:05.383272 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:05.383928 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.383942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.385518 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.385526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.387455 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.387473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.388972 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.388984 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.389320 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.390130 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:05.390159 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:05.391626 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:05.395819 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.395826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.395830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.396448 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.396962 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.396966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.396969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.397157 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.397961 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:05.397970 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:05.398216 (+ 246us) client_negotiation.cc:769] Sending connection context
0923 21:01:05.399649 (+ 1433us) client_negotiation.cc:240] Negotiation successful
0923 21:01:05.400060 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":221,"spinlock_wait_cycles":39808,"thread_start_us":428,"threads_started":1}
I20260923 21:01:05.401110 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.380028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.0.1:53920)
0923 21:01:05.381630 (+ 1602us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:05.381654 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:05.381871 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:05.382429 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:05.382434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:05.383268 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:05.383942 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.383956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.385562 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.385568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.387534 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.387552 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.388929 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.388939 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.389139 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.389943 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:05.389971 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:05.391391 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:05.395843 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.395848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.395852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.396595 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.397004 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.397009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.397012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.397244 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.398915 (+ 1671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:05.398920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:05.399062 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 21:01:05.400266 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0923 21:01:05.400571 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1443,"mutex_wait_us":66,"thread_start_us":565,"threads_started":1}
Sep 23 21:01:05 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260923 21:01:05.431354 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.408324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.0.97.61:33323)
0923 21:01:05.408530 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.408539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.411210 (+ 2671us) server_negotiation.cc:407] Connection header received
0923 21:01:05.411645 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.411652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.411779 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.411900 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:05.413587 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.414786 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.417103 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.417873 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.418016 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.424204 (+ 6188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:05.424211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:05.424216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:05.424305 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:05.427612 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.428716 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.428721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.428723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.428996 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.429482 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.429487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.429490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.429973 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:05.430136 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.430695 (+ 559us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.430954 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:01:05.431408 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.408305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.0.1:53924)
0923 21:01:05.410981 (+ 2676us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:05.411096 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:05.411375 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:05.411992 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:05.412004 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:05.413040 (+ 1036us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:05.413379 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.413398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.414957 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.414962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.416912 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.416932 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.418094 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.418100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.418395 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.419269 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:05.419308 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:05.423825 (+ 4517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:05.427811 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.427832 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.427841 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.428571 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.429129 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.429134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.429137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.429331 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.430119 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:05.430128 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:05.430374 (+ 246us) client_negotiation.cc:769] Sending connection context
0923 21:01:05.430590 (+ 216us) client_negotiation.cc:240] Negotiation successful
0923 21:01:05.430964 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2351,"thread_start_us":2191,"threads_started":1}
I20260923 21:01:05.431380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46465
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:01:05.435001 6549 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:33323: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f90374f37df945d1805e836834117852"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 33323
}
attrs {
promote: true
}
}
dest_uuid: "709239c2e0d0464882bf6056e5b9c0b3"
cas_config_opid_index: -1
I20260923 21:01:05.436087 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } }
I20260923 21:01:05.442057 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f90374f37df945d1805e836834117852 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } }
I20260923 21:01:05.443397 7054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:05.446756 7110 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:33323
I20260923 21:01:05.447633 7012 master.cc:537] Master@127.0.97.61:33323 shutting down...
I20260923 21:01:05.447458 7110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } }
I20260923 21:01:05.447778 7110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:05.449523 7054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Refusing update from remote peer 709239c2e0d0464882bf6056e5b9c0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 21:01:05.451071 7110 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 21:01:05.455276 6533 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 21:01:05.458880 6533 consensus_peers.cc:487] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 -> Peer f90374f37df945d1805e836834117852 (127.0.97.61:33323): Couldn't send request to peer f90374f37df945d1805e836834117852. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 21:01:05.461285 7012 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:05.461827 7012 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:05.462075 7012 tablet_replica.cc:330] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: stopping tablet replica
I20260923 21:01:05.470506 7012 master.cc:552] Master@127.0.97.61:33323 shutdown complete.
I20260923 21:01:05.470695 7012 master_runner.cc:305] Clearing existing system tablet
I20260923 21:01:05.471288 7012 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:05.471467 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:05.474721 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 21:01:05.476708 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:05.477123 7012 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:05.477581 7012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 21:01:05.477723 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
uuid: "f90374f37df945d1805e836834117852"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "0ead0027c67d787ee103029e8c999c16"
server_key_iv: "e6dee8e1c2fca0cb97b6752063c73ab9"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.483067 7012 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:05.491801 7012 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 21:01:05.492172 7012 log.cc:1196] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 21:01:05.492776 7012 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Deleting consensus metadata
I20260923 21:01:05.493144 7012 master_runner.cc:315] Copying system tablet from 127.0.97.62:39963
I20260923 21:01:05.496206 7012 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39963
I20260923 21:01:05.519291 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.497186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48028)
0923 21:01:05.503019 (+ 5833us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:05.503052 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:05.503342 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:05.503904 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:05.503910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:05.504520 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:05.504811 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.504824 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.506595 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.506601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.508188 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.508210 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.509353 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.509360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.509564 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.510477 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:05.510502 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:05.511617 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:05.516061 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.516072 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.516076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.516648 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.517139 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:05.517145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:05.517149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:05.517323 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:05.518047 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:05.518053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:05.518248 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 21:01:05.518440 (+ 192us) client_negotiation.cc:240] Negotiation successful
0923 21:01:05.518791 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5608,"thread_start_us":5461,"threads_started":1}
I20260923 21:01:05.520479 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.497193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.0.97.62:39963)
0923 21:01:05.497381 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.497387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.503200 (+ 5813us) server_negotiation.cc:407] Connection header received
0923 21:01:05.503547 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.503555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.503694 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.503852 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:05.505029 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.506396 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.508405 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.509101 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.509327 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.511901 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:05.511908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:05.511913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:05.511955 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:05.515841 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.516819 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.516825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.516832 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.516993 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:05.517462 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:05.517467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:05.517471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:05.517835 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:05.517986 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.519837 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.520092 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:01:05.521955 6569 tablet_copy_service.cc:140] P 709239c2e0d0464882bf6056e5b9c0b3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f90374f37df945d1805e836834117852 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48028)
I20260923 21:01:05.522233 6569 tablet_copy_service.cc:161] P 709239c2e0d0464882bf6056e5b9c0b3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f90374f37df945d1805e836834117852 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48028: session id = f90374f37df945d1805e836834117852-00000000000000000000000000000000
I20260923 21:01:05.527107 6569 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 21:01:05.529650 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 21:01:05.537000 7012 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: tablet copy: Starting download of 0 data blocks...
I20260923 21:01:05.537380 7012 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: tablet copy: Starting download of 1 WAL segments...
I20260923 21:01:05.540998 7012 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 21:01:05.545990 6569 tablet_copy_service.cc:342] P 709239c2e0d0464882bf6056e5b9c0b3: Request end of tablet copy session f90374f37df945d1805e836834117852-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48028
I20260923 21:01:05.546301 6569 tablet_copy_service.cc:433] P 709239c2e0d0464882bf6056e5b9c0b3: ending tablet copy session f90374f37df945d1805e836834117852-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f90374f37df945d1805e836834117852
I20260923 21:01:05.559214 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:05.562562 7122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:05.568389 7123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:05.576093 7012 server_base.cc:1034] running on GCE node
W20260923 21:01:05.576305 7125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:05.576826 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:05.577564 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:05.578805 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790197265578738 us; error 34 us; skew 500 ppm
I20260923 21:01:05.579057 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:05 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:05.581498 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260923 21:01:05.582852 7012 webserver.cc:460] Webserver started at http://127.0.97.61:40655/ using document root <none> and password file <none>
I20260923 21:01:05.583235 7012 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:05.583318 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:05.586609 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 21:01:05.588796 7132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:05.589318 7012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260923 21:01:05.589538 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
uuid: "f90374f37df945d1805e836834117852"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "0ead0027c67d787ee103029e8c999c16"
server_key_iv: "e6dee8e1c2fca0cb97b6752063c73ab9"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.589823 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:05.634400 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:05.638962 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:05.688913 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33323
I20260923 21:01:05.689077 7183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33323 every 8 connection(s)
I20260923 21:01:05.690297 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
I20260923 21:01:05.697356 7184 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:01:05.702075 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Bootstrap starting.
I20260923 21:01:05.724089 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:01:05.724694 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Bootstrap complete.
I20260923 21:01:05.726138 7184 raft_consensus.cc:348] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } }
I20260923 21:01:05.726465 7184 raft_consensus.cc:729] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f90374f37df945d1805e836834117852, State: Initialized, Role: LEARNER
I20260923 21:01:05.726732 7184 consensus_queue.cc:260] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } }
I20260923 21:01:05.728092 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } }
I20260923 21:01:05.728740 7184 sys_catalog.cc:564] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:05.728750 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: LEARNER
I20260923 21:01:05.739584 7198 catalog_manager.cc:1259] Loaded cluster ID: 6b5f8a05b56b494e9540ffb2aa37be8d
I20260923 21:01:05.739744 7198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: loading cluster ID for follower catalog manager: success
I20260923 21:01:05.744026 7198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: acquiring CA information for follower catalog manager: success
I20260923 21:01:05.747689 7198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 21:01:05.748865 7006 heartbeater.cc:503] Master 127.0.97.62:39963 was elected leader, sending a full tablet report...
W20260923 21:01:05.878233 7109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:05.878714 7109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:05.878870 7109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:05.887099 7109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:05.887270 7109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:05.887384 7109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:05.887486 7109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:05.887569 7109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:05.899168 7109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:05.901062 7109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:05.902771 7109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:05.916538 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:05.916505 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:05.916450 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:05.917526 7109 server_base.cc:1034] running on GCE node
I20260923 21:01:05.918473 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:05.920498 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:05.921742 7109 hybrid_clock.cc:648] HybridClock initialized: now 1790197265921663 us; error 68 us; skew 500 ppm
I20260923 21:01:05.922098 7109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:05 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:05.925937 7109 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260923 21:01:05.927973 7109 webserver.cc:460] Webserver started at http://127.0.97.60:37863/ using document root <none> and password file <none>
I20260923 21:01:05.928805 7109 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:05.928970 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:05.929311 7109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:05.932881 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/instance:
uuid: "a1a59587862b4f15be79496c93feb622"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "ff94042b21f4710dee1c5d40d94da408"
server_key_iv: "442aa652a3f36d104761796057604312"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.934080 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal/instance:
uuid: "a1a59587862b4f15be79496c93feb622"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "ff94042b21f4710dee1c5d40d94da408"
server_key_iv: "442aa652a3f36d104761796057604312"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.940884 7109 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:01:05.945401 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:05.945788 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.935087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.0.97.61:33323)
0923 21:01:05.938869 (+ 3782us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:05.938873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:05.938897 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:01:05.938960 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:05.938964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:05.939043 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:05.939226 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:05.939734 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.940745 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.943181 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.944408 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.944555 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.944807 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:05.945255 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 21:01:05.945418 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3663,"thread_start_us":3490,"threads_started":1}
I20260923 21:01:05.945858 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:05.934898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.0.1:53930)
0923 21:01:05.935489 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:05.935524 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:05.935696 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:05.939275 (+ 3579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:05.939281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:05.939318 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:05.939548 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.939566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.940914 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.940921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.942988 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:05.943005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:05.944680 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:05.944692 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:05.945006 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:05.945081 (+ 75us) client_negotiation.cc:769] Sending connection context
0923 21:01:05.945220 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 21:01:05.945392 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":297,"threads_started":1}
I20260923 21:01:05.947242 7159 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 21:01:05.947675 7109 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:01:05.947865 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
uuid: "a1a59587862b4f15be79496c93feb622"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "ff94042b21f4710dee1c5d40d94da408"
server_key_iv: "442aa652a3f36d104761796057604312"
server_key_version: "encryptionkey@0"
I20260923 21:01:05.948045 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709239c2e0d0464882bf6056e5b9c0b3. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } }
I20260923 21:01:05.948307 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: LEARNER
I20260923 21:01:05.948132 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:05.978317 7109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:05.984061 7109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:05.984382 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:06.007251 7109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46465
I20260923 21:01:06.007387 7266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46465 every 8 connection(s)
I20260923 21:01:06.008807 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
I20260923 21:01:06.013592 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:06.014097 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 7109
I20260923 21:01:06.024873 7267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:06.045425 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.016777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.0.97.60:46465)
0923 21:01:06.017873 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.017899 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.018264 (+ 365us) server_negotiation.cc:407] Connection header received
0923 21:01:06.021078 (+ 2814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.021117 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.021872 (+ 755us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.022484 (+ 612us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.023990 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.026274 (+ 2284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.028182 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.029872 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.030104 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.035898 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.035908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.035923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.035975 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.040128 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.041504 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.041514 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.041522 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.041791 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.042317 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.042376 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.042380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.042990 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.043321 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.043831 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.044333 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":493,"threads_started":1}
I20260923 21:01:06.046877 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.028425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.0.97.62:39963)
0923 21:01:06.029455 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.029461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.030216 (+ 755us) server_negotiation.cc:407] Connection header received
0923 21:01:06.030281 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.030289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.030407 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.030496 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.032134 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.032933 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.035021 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.035769 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.035926 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.040323 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.040331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.040335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.040385 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.043862 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.044657 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.044663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.044668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.044795 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.045201 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.045213 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.045215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.045527 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.045707 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.046246 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.046494 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":744,"threads_started":1}
I20260923 21:01:06.046983 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.028421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48034)
0923 21:01:06.029883 (+ 1462us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.029928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.030102 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.030620 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.030629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.031571 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.031958 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.031981 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.033114 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.033123 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.034857 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.034871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.035978 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.035985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.036208 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.036919 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.036946 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.040038 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.044007 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.044021 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.044028 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.044517 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.044909 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.044914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.044918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.045070 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.045768 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.045774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.045922 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.046134 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.046494 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"thread_start_us":1031,"threads_started":1}
I20260923 21:01:06.048972 7267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260923 21:01:06.064157 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.050094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.0.97.61:33323)
0923 21:01:06.050415 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.050422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.050603 (+ 181us) server_negotiation.cc:407] Connection header received
0923 21:01:06.050800 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.050805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.050879 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.051000 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.051833 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.052559 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.053958 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.054584 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.054724 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.058162 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.058168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.058172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.058209 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.061323 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.062132 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.062138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.062144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.062284 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.062710 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.062716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.062720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.063006 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.063166 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.063561 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.063789 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 21:01:06.064142 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.050212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.0.1:53940)
0923 21:01:06.050500 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.050526 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.050674 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.051018 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.051022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.051432 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.051690 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.051699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.052701 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.052706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.053779 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.053791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.054738 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.054744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.054916 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.055696 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.055724 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.057917 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.061538 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.061544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.061548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.061971 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.062449 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.062454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.062457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.062595 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.063144 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.063148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.063306 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.063447 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.063779 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260923 21:01:06.065687 7267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197265, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 21:01:06.080391 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.066453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.0.1:35256)
0923 21:01:06.066747 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.066764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.066860 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.067221 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.067224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.067562 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.067784 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.067814 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.068769 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.068777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.070025 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.070037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.070819 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.070824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.071033 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.071709 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.071731 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.074160 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.077816 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.077821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.077824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.078305 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.078826 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.078830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.078832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.078997 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.079561 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.079564 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.079704 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.079827 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.080062 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 21:01:06.080762 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.066520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.0.97.60:46465)
0923 21:01:06.066743 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.066748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.066808 (+ 60us) server_negotiation.cc:407] Connection header received
0923 21:01:06.066999 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.067003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.067064 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.067163 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.067947 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.068646 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.070183 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.070648 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.070796 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.074719 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.074725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.074729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.074766 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.077639 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.078503 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.078508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.078511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.078713 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.079127 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.079131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.079133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.079450 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.079666 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.079919 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.080245 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":133}
I20260923 21:01:06.087164 7267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Bootstrap starting.
I20260923 21:01:06.091152 7267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:06.092386 7267 log.cc:824] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:06.096323 7267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: No bootstrap required, opened a new log
I20260923 21:01:06.104355 7267 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } }
I20260923 21:01:06.104737 7267 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:06.104862 7267 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1a59587862b4f15be79496c93feb622, State: Initialized, Role: FOLLOWER
I20260923 21:01:06.105577 7267 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } }
I20260923 21:01:06.106760 7274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } } }
I20260923 21:01:06.107165 7274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:01:06.107765 7267 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 21:01:06.117852 7285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 21:01:06.118033 7285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 21:01:06.123831 7109 master_runner.cc:428] Detected that this master a1a59587862b4f15be79496c93feb622 is joining an existing cluster
I20260923 21:01:06.124014 7109 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:46465
I20260923 21:01:06.142143 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.128964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.0.97.62:39963)
0923 21:01:06.129156 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.129163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.129296 (+ 133us) server_negotiation.cc:407] Connection header received
0923 21:01:06.129546 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.129551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.129628 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.129734 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.130862 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.131600 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.133762 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.134186 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.134329 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.136357 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.136370 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.136374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.136408 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.139364 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.140012 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.140017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.140021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.140205 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.140632 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.140637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.140640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.140932 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.141102 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.141576 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.141805 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 21:01:06.142668 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.128406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48044)
0923 21:01:06.129175 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.129198 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.129405 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.129765 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.129769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.130309 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.130705 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.130715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.131800 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.131806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.133608 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.133621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.134426 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.134434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.134602 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.135326 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.135344 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.136141 (+ 797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.139509 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.139514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.139518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.139882 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.140322 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.140327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.140330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.140487 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.141052 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.141057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.141277 (+ 220us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.141466 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.142143 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":607,"threads_started":1}
I20260923 21:01:06.152194 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 6610
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 21:01:06.166613 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.139126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.0.97.60:46465)
0923 21:01:06.139320 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.139326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.143115 (+ 3789us) server_negotiation.cc:407] Connection header received
0923 21:01:06.143302 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.143307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.143366 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.143425 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.144656 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.145691 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.148571 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.149411 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.149525 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.152946 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.152953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.152959 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.152997 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.159811 (+ 6814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.163456 (+ 3645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.163462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.163465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.163621 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.164008 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.164014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.164029 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.164472 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.164661 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.165536 (+ 875us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.165982 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":183,"server-negotiator.queue_time_us":54}
I20260923 21:01:06.169414 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.128613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.0.1:53952)
0923 21:01:06.138926 (+ 10313us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.138949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.145825 (+ 6876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.146467 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.146473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.147728 (+ 1255us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.148087 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.148104 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.149434 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.149442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.154301 (+ 4859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.154316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.156097 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.156109 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.159958 (+ 3849us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.160819 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.160847 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.161999 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.166021 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.166025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.166028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.167036 (+ 1008us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.167461 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.167466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.167469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.167645 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.168371 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.168376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.168552 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.168678 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.168976 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10128,"mutex_wait_us":1118,"thread_start_us":9885,"threads_started":1}
I20260923 21:01:06.170440 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.128409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.97.61:33323)
0923 21:01:06.128625 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.128634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.139044 (+ 10410us) server_negotiation.cc:407] Connection header received
0923 21:01:06.146024 (+ 6980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.146033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.146138 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.146240 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.148292 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.149251 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.154546 (+ 5295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.155186 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.155280 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.162252 (+ 6972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.162260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.162265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.162310 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.165863 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.167202 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.167212 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.167214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.167333 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.167788 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.167793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.167796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.168141 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.168278 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.169855 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.170092 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260923 21:01:06.173408 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.139039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.0.1:35260)
0923 21:01:06.143047 (+ 4008us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.143065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.143190 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.143624 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.143628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.144264 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.144501 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.144512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.146125 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.146132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.148408 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.148425 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.149877 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.149883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.150571 (+ 688us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.151171 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.151191 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.152668 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.161441 (+ 8773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.161446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.161451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.163321 (+ 1870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.163743 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.163748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.163752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.163890 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.164716 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.164720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.164898 (+ 178us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.165077 (+ 179us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.166468 (+ 1391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3912,"mutex_wait_us":408,"thread_start_us":6484,"threads_started":1}
I20260923 21:01:06.180404 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.149407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.0.1:35266)
0923 21:01:06.149599 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.149625 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.149737 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.150955 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.150961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.151501 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.151727 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.151742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.156876 (+ 5134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.156885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.158454 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.158477 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.159869 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.159877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.160079 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.161036 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.161064 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.164287 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.176877 (+ 12590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.176884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.176888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.177526 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.178009 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.178014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.178024 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.178149 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.178925 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.178933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.179128 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.179304 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.179637 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 21:01:06.180516 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.149776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.0.97.60:46465)
0923 21:01:06.150529 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.150535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.150559 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:01:06.150664 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.150670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.150762 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.151889 (+ 1127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.152280 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.156656 (+ 4376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.158792 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.159647 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.164492 (+ 4845us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.164763 (+ 271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.164770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.164779 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.164813 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.176679 (+ 11866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.177680 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.177685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.177689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.177864 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.178271 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.178277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.178280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.178757 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.178917 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.179436 (+ 519us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.179685 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":392,"threads_started":1}
I20260923 21:01:06.182297 6549 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:46465: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a1a59587862b4f15be79496c93feb622"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 46465
}
attrs {
promote: true
}
}
dest_uuid: "709239c2e0d0464882bf6056e5b9c0b3"
cas_config_opid_index: 5
I20260923 21:01:06.182937 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.153488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.0.97.1:38547)
0923 21:01:06.155083 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.155091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.155119 (+ 28us) server_negotiation.cc:407] Connection header received
0923 21:01:06.155175 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.155183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.155422 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.155700 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.157383 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.159477 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.161947 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.162954 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.166818 (+ 3864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.172928 (+ 6110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.172953 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.172982 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.173024 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.177682 (+ 4658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.178708 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.178720 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.178723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.178949 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.179467 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.179472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.179475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.179910 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.180191 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.182014 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.182376 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1432,"thread_start_us":1183,"threads_started":1}
I20260923 21:01:06.183602 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } }
I20260923 21:01:06.184995 6733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:06.185508 7111 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a1a59587862b4f15be79496c93feb622 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.190168 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 LEARNER]: Refusing update from remote peer 709239c2e0d0464882bf6056e5b9c0b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 21:01:06.191974 7242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:06.198948 7111 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:46465
I20260923 21:01:06.199905 7109 master.cc:537] Master@127.0.97.60:46465 shutting down...
I20260923 21:01:06.232256 7242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Refusing update from remote peer 709239c2e0d0464882bf6056e5b9c0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260923 21:01:06.241218 7221 tablet.cc:2369] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:06.257603 7109 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:06.258234 7109 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:06.258530 7109 tablet_replica.cc:330] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: stopping tablet replica
I20260923 21:01:06.268373 7299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260923 21:01:06.273128 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a1a59587862b4f15be79496c93feb622 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.283241 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.283640 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: LEARNER
I20260923 21:01:06.289723 7299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: true } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.290355 7299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:06.290450 7109 master.cc:552] Master@127.0.97.60:46465 shutdown complete.
I20260923 21:01:06.290542 7109 master_runner.cc:305] Clearing existing system tablet
I20260923 21:01:06.290129 7296 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.004s
I20260923 21:01:06.291005 7109 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:06.291162 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:06.291596 7111 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: attempting to promote NON_VOTER f90374f37df945d1805e836834117852 to VOTER
I20260923 21:01:06.292474 7111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } }
I20260923 21:01:06.296685 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.295956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.0.1:35270)
0923 21:01:06.296220 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.296417 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 21:01:06.298739 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.298220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.0.1:35278)
0923 21:01:06.298435 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.298538 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 21:01:06.299067 6533 consensus_peers.cc:487] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 -> Peer a1a59587862b4f15be79496c93feb622 (127.0.97.60:46465): Couldn't send request to peer a1a59587862b4f15be79496c93feb622. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 21:01:06.300524 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 LEARNER]: Refusing update from remote peer 709239c2e0d0464882bf6056e5b9c0b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 21:01:06.301481 7299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 21:01:06.310835 7109 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.000s sys 0.008s
I20260923 21:01:06.310729 7298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f90374f37df945d1805e836834117852 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.316942 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.317274 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:06.317629 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.317844 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:06.318403 7305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:06.318199 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f90374f37df945d1805e836834117852 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.321377 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.321655 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:01:06.321594 7109 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:06.322204 7109 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.001s
I20260923 21:01:06.322414 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
uuid: "a1a59587862b4f15be79496c93feb622"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "ff94042b21f4710dee1c5d40d94da408"
server_key_iv: "442aa652a3f36d104761796057604312"
server_key_version: "encryptionkey@0"
I20260923 21:01:06.326309 7109 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:06.332283 7109 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 21:01:06.332479 7109 log.cc:1196] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 21:01:06.332993 7109 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Deleting consensus metadata
I20260923 21:01:06.333319 7109 master_runner.cc:315] Copying system tablet from 127.0.97.62:39963
I20260923 21:01:06.340152 7109 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39963
I20260923 21:01:06.353487 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.341059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.97.62:39963)
0923 21:01:06.341337 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.341344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.341807 (+ 463us) server_negotiation.cc:407] Connection header received
0923 21:01:06.342019 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.342025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.342115 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.342230 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:06.343132 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.343882 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.345593 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.346038 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.346181 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.348135 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:06.348141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:06.348148 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:06.348187 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:06.350875 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.351552 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.351559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.351562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.351702 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:06.352079 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:06.352084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:06.352089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:06.352354 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:06.352508 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.352895 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.353104 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260923 21:01:06.353554 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.341023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48048)
0923 21:01:06.341685 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.341707 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.341888 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.342261 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.342266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.342750 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:06.342972 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.342982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.344023 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.344029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.345428 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.345441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.346214 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.346220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.346445 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.347176 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:06.347198 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:06.347931 (+ 733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:06.351031 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.351036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.351040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.351431 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.351815 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:06.351821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:06.351824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:06.351970 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:06.352473 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:06.352478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:06.352639 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.352790 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.353121 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":367,"threads_started":1}
I20260923 21:01:06.354643 6569 tablet_copy_service.cc:140] P 709239c2e0d0464882bf6056e5b9c0b3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a1a59587862b4f15be79496c93feb622 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:48048)
I20260923 21:01:06.354826 6569 tablet_copy_service.cc:161] P 709239c2e0d0464882bf6056e5b9c0b3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a1a59587862b4f15be79496c93feb622 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:48048: session id = a1a59587862b4f15be79496c93feb622-00000000000000000000000000000000
I20260923 21:01:06.357183 6569 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 21:01:06.358950 7109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 21:01:06.364444 7109 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: tablet copy: Starting download of 0 data blocks...
I20260923 21:01:06.364773 7109 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: tablet copy: Starting download of 1 WAL segments...
I20260923 21:01:06.367182 7109 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 21:01:06.371150 6569 tablet_copy_service.cc:342] P 709239c2e0d0464882bf6056e5b9c0b3: Request end of tablet copy session a1a59587862b4f15be79496c93feb622-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:48048
I20260923 21:01:06.371320 6569 tablet_copy_service.cc:433] P 709239c2e0d0464882bf6056e5b9c0b3: ending tablet copy session a1a59587862b4f15be79496c93feb622-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a1a59587862b4f15be79496c93feb622
I20260923 21:01:06.381879 7109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:06.384793 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:06.388211 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:06.402556 7109 server_base.cc:1034] running on GCE node
W20260923 21:01:06.404028 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:06.404724 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:06.405473 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:06.406713 7109 hybrid_clock.cc:648] HybridClock initialized: now 1790197266406662 us; error 39 us; skew 500 ppm
I20260923 21:01:06.406991 7109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:06.409461 7109 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260923 21:01:06.411042 7109 webserver.cc:460] Webserver started at http://127.0.97.60:42113/ using document root <none> and password file <none>
I20260923 21:01:06.411365 7109 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:06.411487 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:06.413905 7109 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 21:01:06.415789 7322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:06.416260 7109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 21:01:06.416390 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
uuid: "a1a59587862b4f15be79496c93feb622"
format_stamp: "Formatted at 2026-09-23 21:01:05 on dist-test-slave-8xn5"
server_key: "ff94042b21f4710dee1c5d40d94da408"
server_key_iv: "442aa652a3f36d104761796057604312"
server_key_version: "encryptionkey@0"
I20260923 21:01:06.416584 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:06.448355 7109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:06.450520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 6610
I20260923 21:01:06.453286 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:06.473547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38547
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39963
--webserver_port=35359
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:01:06.495054 7109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46465
I20260923 21:01:06.495150 7374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46465 every 8 connection(s)
I20260923 21:01:06.496747 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
I20260923 21:01:06.501624 7375 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:01:06.505617 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Bootstrap starting.
I20260923 21:01:06.525233 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:01:06.525966 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: Bootstrap complete.
I20260923 21:01:06.527153 7375 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } }
I20260923 21:01:06.527464 7375 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a1a59587862b4f15be79496c93feb622, State: Initialized, Role: LEARNER
I20260923 21:01:06.527763 7375 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } }
I20260923 21:01:06.528702 7378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.529063 7375 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:06.529230 7378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: This master's current role is: LEARNER
I20260923 21:01:06.538525 7389 catalog_manager.cc:1259] Loaded cluster ID: 6b5f8a05b56b494e9540ffb2aa37be8d
I20260923 21:01:06.538671 7389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: loading cluster ID for follower catalog manager: success
I20260923 21:01:06.542290 7389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: acquiring CA information for follower catalog manager: success
I20260923 21:01:06.545009 7389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 21:01:06.805250 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.796923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.0.1:35286)
0923 21:01:06.797183 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:06.797211 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:06.797375 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:06.798402 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:06.798408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:06.798448 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:06.798685 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.798699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.800500 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.800507 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.802605 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:06.802632 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.804176 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.804184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:06.804451 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.804499 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 21:01:06.804636 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 21:01:06.804848 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 21:01:06.805315 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:06.797169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.0.97.60:46465)
0923 21:01:06.797958 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:06.797970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:06.798007 (+ 37us) server_negotiation.cc:407] Connection header received
0923 21:01:06.798115 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:06.798120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:06.798209 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:06.798377 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:06.798915 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.799941 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.802874 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:06.803920 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:06.804068 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:06.804243 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:06.804885 (+ 642us) server_negotiation.cc:299] Negotiation successful
0923 21:01:06.805022 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":453,"threads_started":1}
I20260923 21:01:06.806638 7350 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 21:01:06.807363 7378 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709239c2e0d0464882bf6056e5b9c0b3. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:06.807747 7378 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: This master's current role is: LEARNER
W20260923 21:01:06.871206 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:06.871529 7345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:06.871634 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:06.879961 7345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:06.880106 7345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:06.880229 7345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:06.890758 7345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35359
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:06.892174 7345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:06.893683 7345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:06.910035 7395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:06.910104 7398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:06.910159 7396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:06.910856 7345 server_base.cc:1034] running on GCE node
I20260923 21:01:06.911664 7345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:06.912832 7345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:06.914095 7345 hybrid_clock.cc:648] HybridClock initialized: now 1790197266914029 us; error 76 us; skew 500 ppm
I20260923 21:01:06.914595 7345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:06 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:06.918265 7345 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 21:01:06.920121 7345 webserver.cc:460] Webserver started at http://127.0.97.1:35359/ using document root <none> and password file <none>
I20260923 21:01:06.920897 7345 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:06.921532 7345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:06.927919 7345 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:01:06.931634 7405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:06.933557 7345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:01:06.933856 7345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "cdba6177147a47e693ad2d152127f63a"
format_stamp: "Formatted at 2026-09-23 21:01:03 on dist-test-slave-8xn5"
server_key: "4569643d76b8c4dbdbc0dca4117b0d0b"
server_key_iv: "2d79b1171246e07e4e205799952b97fa"
server_key_version: "encryptionkey@0"
I20260923 21:01:06.934988 7345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:06.954216 7345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:06.959817 7345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:06.960146 7345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:06.961382 7345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:06.963073 7345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:06.963173 7345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:06.963336 7345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:06.963424 7345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:07.003902 7345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38547
I20260923 21:01:07.003937 7517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38547 every 8 connection(s)
I20260923 21:01:07.005551 7345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:07.011825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 7345
I20260923 21:01:07.012246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 6744
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197266, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 21:01:07.057731 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.013669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.0.97.2:41743)
0923 21:01:07.029899 (+ 16230us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.029915 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.030821 (+ 906us) server_negotiation.cc:407] Connection header received
0923 21:01:07.030904 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.030913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.031200 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.031357 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.033967 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.035611 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.038512 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.039494 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.039661 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.047007 (+ 7346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.047022 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.047037 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.047081 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.053005 (+ 5924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.054123 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.054135 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.054139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.054380 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.054939 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.054943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.054946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.055463 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.055756 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.057002 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.057294 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15483,"thread_start_us":15280,"threads_started":1}
I20260923 21:01:07.059401 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.010055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59889 (local address 127.0.97.62:39963)
0923 21:01:07.018796 (+ 8741us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.018801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.032296 (+ 13495us) server_negotiation.cc:407] Connection header received
0923 21:01:07.035200 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.035206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.035288 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.035365 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.038537 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.039623 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.043197 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.043934 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.044081 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.050565 (+ 6484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.050574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.050579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.050646 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.054324 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.055501 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.055510 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.055517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.055695 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.056353 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.056358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.056361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.056722 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.056869 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.058805 (+ 1936us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.059048 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8620,"thread_start_us":8482,"threads_started":1}
I20260923 21:01:07.059444 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.009615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45103 (local address 127.0.97.61:33323)
0923 21:01:07.019080 (+ 9465us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.019084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.031473 (+ 12389us) server_negotiation.cc:407] Connection header received
0923 21:01:07.035227 (+ 3754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.035233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.035327 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.035402 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.038536 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.039739 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.043185 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.043897 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.044012 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.050183 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.050190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.050194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.050233 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.055556 (+ 5323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.056460 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.056466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.056470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.057067 (+ 597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.057598 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.057603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.057606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.057940 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.058099 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.058781 (+ 682us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.059036 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9363,"thread_start_us":9462,"threads_started":1}
I20260923 21:01:07.059742 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:07.060478 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.032252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52243 (local address 127.0.97.60:46465)
0923 21:01:07.032482 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.032495 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.033490 (+ 995us) server_negotiation.cc:407] Connection header received
0923 21:01:07.035172 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.035180 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.035271 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.035341 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.038536 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.039718 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.043185 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.043897 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.044012 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.051418 (+ 7406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.051428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.051436 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.051487 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.055388 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.056460 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.056466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.056470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.057065 (+ 595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.057614 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.057619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.057622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.057946 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.058095 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.058781 (+ 686us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.059038 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260923 21:01:07.061833 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.010542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:59889)
0923 21:01:07.032168 (+ 21626us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:07.032193 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:07.034807 (+ 2614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:07.035951 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:07.035964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:07.036984 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:07.038278 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.038306 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.039930 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.039935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.042954 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.042975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.044485 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.044494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.045170 (+ 676us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.046159 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:07.046205 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:07.050254 (+ 4049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:07.054549 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.054560 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.054583 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.055341 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.055871 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.055876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.055880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.056206 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.057137 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:07.057145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:07.057661 (+ 516us) client_negotiation.cc:769] Sending connection context
0923 21:01:07.059680 (+ 2019us) client_negotiation.cc:240] Negotiation successful
0923 21:01:07.060348 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21444,"mutex_wait_us":2956,"spinlock_wait_cycles":19072,"thread_start_us":18910,"threads_started":1}
I20260923 21:01:07.061901 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.010543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.97.1:45103)
0923 21:01:07.031232 (+ 20689us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:07.031314 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:07.034829 (+ 3515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:07.035951 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:07.035964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:07.036994 (+ 1030us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:07.038293 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.038307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.039955 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.039960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.042954 (+ 2994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.042975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.044462 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.044471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.044983 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.046132 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:07.046179 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:07.049805 (+ 3626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:07.055708 (+ 5903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.055712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.055716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.056307 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.057228 (+ 921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.057233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.057237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.057456 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.058181 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:07.058186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:07.058432 (+ 246us) client_negotiation.cc:769] Sending connection context
0923 21:01:07.059733 (+ 1301us) client_negotiation.cc:240] Negotiation successful
0923 21:01:07.060513 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20215,"mutex_wait_us":924,"spinlock_wait_cycles":57600,"thread_start_us":19879,"threads_started":1}
I20260923 21:01:07.063560 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.032641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.97.1:52243)
0923 21:01:07.033340 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:07.033365 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:07.034857 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:07.035951 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:07.035964 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:07.036996 (+ 1032us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:07.038279 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.038307 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.039965 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.039969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.042954 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.042975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.044462 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.044471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.044975 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.046132 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:07.046180 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:07.051129 (+ 4949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:07.055582 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.055588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.055593 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.056307 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.057228 (+ 921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.057233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.057237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.057456 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.058181 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:07.058186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:07.058432 (+ 246us) client_negotiation.cc:769] Sending connection context
0923 21:01:07.062428 (+ 3996us) client_negotiation.cc:240] Negotiation successful
0923 21:01:07.062847 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":627,"spinlock_wait_cycles":29824,"thread_start_us":357,"threads_started":1}
I20260923 21:01:07.063697 7519 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39963
I20260923 21:01:07.063921 7520 heartbeater.cc:347] Connected to a master server at 127.0.97.61:33323
I20260923 21:01:07.064069 7520 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:07.064074 7519 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:07.065138 7519 heartbeater.cc:511] Master 127.0.97.62:39963 requested a full tablet report, sending...
I20260923 21:01:07.065305 7520 heartbeater.cc:511] Master 127.0.97.61:33323 requested a full tablet report, sending...
I20260923 21:01:07.065927 7518 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46465
I20260923 21:01:07.066071 7518 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:07.066555 7518 heartbeater.cc:511] Master 127.0.97.60:46465 requested a full tablet report, sending...
I20260923 21:01:07.067884 6549 ts_manager.cc:194] Re-registered known tserver with Master: cdba6177147a47e693ad2d152127f63a (127.0.97.1:38547)
I20260923 21:01:07.068012 7149 ts_manager.cc:194] Registered new tserver with Master: cdba6177147a47e693ad2d152127f63a (127.0.97.1:38547)
I20260923 21:01:07.069022 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59889
I20260923 21:01:07.070636 7339 ts_manager.cc:194] Registered new tserver with Master: cdba6177147a47e693ad2d152127f63a (127.0.97.1:38547)
I20260923 21:01:07.156967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 6744
I20260923 21:01:07.174165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41743
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39963
--webserver_port=34669
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:01:07.403234 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: true } } }
I20260923 21:01:07.403708 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:07.405354 7298 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: attempting to promote NON_VOTER a1a59587862b4f15be79496c93feb622 to VOTER
I20260923 21:01:07.405962 7298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:07.408335 7350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 LEARNER]: Refusing update from remote peer 709239c2e0d0464882bf6056e5b9c0b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 21:01:07.408473 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Refusing update from remote peer 709239c2e0d0464882bf6056e5b9c0b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 21:01:07.409092 7532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 21:01:07.409541 7298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 21:01:07.413244 7532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1a59587862b4f15be79496c93feb622 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.413997 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1a59587862b4f15be79496c93feb622 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.418912 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.419418 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:07.419842 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.420240 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:07.420822 7537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.421350 7537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:01:07.428203 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.428524 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:07.429042 7350 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a1a59587862b4f15be79496c93feb622 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.433543 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "709239c2e0d0464882bf6056e5b9c0b3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:07.433976 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 21:01:07.546013 7530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:07.546383 7530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:07.546561 7530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:07.554574 7530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:07.554711 7530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:07.554908 7530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:07.566186 7530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34669
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:07.567680 7530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:07.569104 7530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:07.584314 7543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:07.584466 7546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:07.584350 7544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:07.585235 7530 server_base.cc:1034] running on GCE node
I20260923 21:01:07.586063 7530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:07.587314 7530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:07.588603 7530 hybrid_clock.cc:648] HybridClock initialized: now 1790197267588541 us; error 64 us; skew 500 ppm
I20260923 21:01:07.589087 7530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:07.592978 7530 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 21:01:07.594726 7530 webserver.cc:460] Webserver started at http://127.0.97.2:34669/ using document root <none> and password file <none>
I20260923 21:01:07.595563 7530 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:07.595736 7530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:07.602182 7530 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 21:01:07.605590 7553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:07.607537 7530 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:01:07.607779 7530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "5e5abb06795f44babcdb3f3f4040a426"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "09c019cb27c825452ac48deecc0e592f"
server_key_iv: "6aa19610a5011d72b018ef242d9fafc7"
server_key_version: "encryptionkey@0"
I20260923 21:01:07.608942 7530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:07.630861 7530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:07.636308 7530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:07.636632 7530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:07.637720 7530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:07.639422 7530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:07.639621 7530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:07.639772 7530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:07.639868 7530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:07.677512 7530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41743
I20260923 21:01:07.677564 7665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41743 every 8 connection(s)
I20260923 21:01:07.679106 7530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:07.688606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 7530
I20260923 21:01:07.688932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 6878
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197267, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:07.732543 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.703303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58237 (local address 127.0.97.61:33323)
0923 21:01:07.710487 (+ 7184us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.710492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.710509 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:01:07.710559 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.710563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.710631 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.710764 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.712087 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.713345 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.715432 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.716085 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.716233 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.722984 (+ 6751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.722990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.722993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.723066 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.726928 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.727866 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.727875 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.727879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.728024 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.728466 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.728471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.728475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.729268 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.729420 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.731882 (+ 2462us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.732141 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7009,"thread_start_us":6898,"threads_started":1}
I20260923 21:01:07.732625 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.682443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56585 (local address 127.0.97.60:46465)
0923 21:01:07.691359 (+ 8916us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.691370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.698229 (+ 6859us) server_negotiation.cc:407] Connection header received
0923 21:01:07.703815 (+ 5586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.703826 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.703945 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.704039 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.707452 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.708846 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.712456 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.713442 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.713578 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.722142 (+ 8564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.722150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.722155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.722199 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.725936 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.727122 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.727128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.727132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.727282 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.729367 (+ 2085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.729372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.729376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.729742 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.729909 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.731959 (+ 2050us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.732232 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8744,"thread_start_us":15905,"threads_started":1}
Sep 23 21:01:07 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260923 21:01:07.734263 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.683274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.97.2:56585)
0923 21:01:07.696742 (+ 13468us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:07.696836 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:07.702178 (+ 5342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:07.704712 (+ 2534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:07.704733 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:07.705937 (+ 1204us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:07.707138 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.707173 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.709124 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.709131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.712194 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.712224 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.714190 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.714198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.714605 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.715783 (+ 1178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:07.715830 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:07.721799 (+ 5969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:07.726141 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.726152 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.726174 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.726920 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.727418 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.727425 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.727428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.727735 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.729971 (+ 2236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:07.729976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:07.731086 (+ 1110us) client_negotiation.cc:769] Sending connection context
0923 21:01:07.731214 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 21:01:07.731468 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12964,"thread_start_us":19074,"threads_started":1}
I20260923 21:01:07.735033 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.703292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.97.2:58237)
0923 21:01:07.709844 (+ 6552us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:07.709863 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:07.709978 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:07.710797 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:07.710802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:07.711526 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:07.711890 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.711905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.713537 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.713543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.715197 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.715215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.716527 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.716533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.716759 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.717523 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:07.717547 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:07.722120 (+ 4573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:07.727093 (+ 4973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.727097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.727101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.727711 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.728157 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.728161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.728165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.728338 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.729513 (+ 1175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:07.729522 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:07.730026 (+ 504us) client_negotiation.cc:769] Sending connection context
0923 21:01:07.730692 (+ 666us) client_negotiation.cc:240] Negotiation successful
0923 21:01:07.731306 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6394,"thread_start_us":6214,"threads_started":1}
I20260923 21:01:07.736263 7666 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46465
I20260923 21:01:07.736368 7668 heartbeater.cc:347] Connected to a master server at 127.0.97.61:33323
I20260923 21:01:07.736645 7666 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:07.736649 7668 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:07.738170 7666 heartbeater.cc:511] Master 127.0.97.60:46465 requested a full tablet report, sending...
I20260923 21:01:07.738193 7668 heartbeater.cc:511] Master 127.0.97.61:33323 requested a full tablet report, sending...
I20260923 21:01:07.740363 7339 ts_manager.cc:194] Registered new tserver with Master: 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:07.740410 7149 ts_manager.cc:194] Registered new tserver with Master: 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:07.742513 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.682383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59737 (local address 127.0.97.62:39963)
0923 21:01:07.719868 (+ 37485us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.719877 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.719904 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:01:07.719981 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.719989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.720116 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.721135 (+ 1019us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.722530 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.723562 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.725691 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.726369 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.726510 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.731099 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.731107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.731113 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.731184 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.735365 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.739093 (+ 3728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.739099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.739103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.739298 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.740727 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.740731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.740734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.741071 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.741252 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.741846 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.742077 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37320,"thread_start_us":38492,"threads_started":1}
I20260923 21:01:07.742547 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.683274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:59737)
0923 21:01:07.702976 (+ 19702us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:07.703005 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:07.703191 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:07.721174 (+ 17983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:07.721194 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:07.721995 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:07.722366 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.722385 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.723692 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.723699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.724782 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:07.724797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.726561 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.726567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:07.726754 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.727329 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:07.727378 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:07.730857 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:07.735545 (+ 4688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.735553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.735559 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.736086 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.739447 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:07.739452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:07.739458 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:07.739679 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:07.741232 (+ 1553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:07.741237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:07.741401 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 21:01:07.741718 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 21:01:07.742058 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19432,"mutex_wait_us":56,"thread_start_us":13396,"threads_started":1}
I20260923 21:01:07.743320 7667 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39963
I20260923 21:01:07.743474 7667 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:07.743896 7667 heartbeater.cc:511] Master 127.0.97.62:39963 requested a full tablet report, sending...
I20260923 21:01:07.744217 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:07.691031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.0.97.3:34747)
0923 21:01:07.711410 (+ 20379us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:07.711420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:07.712205 (+ 785us) server_negotiation.cc:407] Connection header received
0923 21:01:07.712285 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:07.712295 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:07.712739 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:07.712920 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:07.720997 (+ 8077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.722707 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.726379 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:07.728266 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:07.728346 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:07.736418 (+ 8072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:07.736439 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:07.736460 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:07.736521 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:07.740262 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.741359 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.741369 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.741372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.741563 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:07.742089 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:07.742094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:07.742097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:07.742552 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:07.742885 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:07.743513 (+ 628us) server_negotiation.cc:299] Negotiation successful
0923 21:01:07.743809 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20187,"thread_start_us":20025,"threads_started":1}
I20260923 21:01:07.745031 6549 ts_manager.cc:194] Re-registered known tserver with Master: 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:07.745851 7001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:07.746037 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59737
I20260923 21:01:07.831956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 6878
W20260923 21:01:07.848237 6536 connection.cc:535] server connection from 127.0.97.3:34181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 21:01:07.848835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34747
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39963
--webserver_port=37765
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:01:08.072194 7519 heartbeater.cc:503] Master 127.0.97.62:39963 was elected leader, sending a full tablet report...
W20260923 21:01:08.210220 7679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:08.210840 7679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:08.211035 7679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:08.221630 7679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:08.221819 7679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:08.222019 7679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:08.233114 7679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37765
--tserver_master_addrs=127.0.97.62:39963,127.0.97.61:33323,127.0.97.60:46465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:08.234711 7679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:08.236223 7679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:08.252720 7685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:08.253270 7687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:08.253587 7684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:08.253537 7679 server_base.cc:1034] running on GCE node
I20260923 21:01:08.254391 7679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:08.255465 7679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:08.256707 7679 hybrid_clock.cc:648] HybridClock initialized: now 1790197268256654 us; error 52 us; skew 500 ppm
I20260923 21:01:08.257041 7679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:08 dist-test-slave-8xn5 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:08.260490 7679 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 21:01:08.262220 7679 webserver.cc:460] Webserver started at http://127.0.97.3:37765/ using document root <none> and password file <none>
I20260923 21:01:08.263088 7679 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:08.263291 7679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:08.269385 7679 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 21:01:08.272630 7694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:08.274268 7679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:01:08.274549 7679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "ddf0c67eda0742cda4369580b7ba83fa"
format_stamp: "Formatted at 2026-09-23 21:01:04 on dist-test-slave-8xn5"
server_key: "dfcd89d27bc6f661a25de71d2a40f7e0"
server_key_iv: "e7333091c5e498ac76af8b129756a6c6"
server_key_version: "encryptionkey@0"
I20260923 21:01:08.275610 7679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:08.303138 7679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:08.308245 7679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:08.308605 7679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:08.309717 7679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:08.311201 7679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:08.311331 7679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:08.311455 7679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:08.311549 7679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:08.348188 7679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34747
I20260923 21:01:08.348248 7806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34747 every 8 connection(s)
I20260923 21:01:08.349767 7679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:08.354955 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 7679
Sep 23 21:01:08 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 23 21:01:08 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 23 21:01:08 dist-test-slave-8xn5 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197268, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260923 21:01:08.406419 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.353979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:39393)
0923 21:01:08.368567 (+ 14588us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.368601 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.376583 (+ 7982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.377686 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.377696 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.379439 (+ 1743us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:08.380444 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.380482 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.384507 (+ 4025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.384511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.386212 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.386227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.387075 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.387082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.387293 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.388259 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:08.388283 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:08.392444 (+ 4161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:08.398889 (+ 6445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:08.398902 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:08.398941 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:08.399909 (+ 968us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:08.400612 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:08.400618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:08.400622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:08.400955 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:08.401829 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:08.401842 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:08.402438 (+ 596us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.402861 (+ 423us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.404205 (+ 1344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14324,"mutex_wait_us":139,"thread_start_us":14548,"threads_started":1}
I20260923 21:01:08.407995 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.353101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39393 (local address 127.0.97.62:39963)
0923 21:01:08.370847 (+ 17746us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.370858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.370897 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:01:08.377145 (+ 6248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.377150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.377227 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.377362 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.381796 (+ 4434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.382806 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.386411 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.386867 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.387043 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.393512 (+ 6469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.393525 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.393531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.393587 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.398650 (+ 5063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.400209 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.400221 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.400229 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.400423 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.401133 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.401141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.401145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.401632 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.407164 (+ 5532us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.407358 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.407638 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17559,"thread_start_us":17434,"threads_started":1}
I20260923 21:01:08.409544 7808 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39963
I20260923 21:01:08.409909 7808 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:08.410938 7808 heartbeater.cc:511] Master 127.0.97.62:39963 requested a full tablet report, sending...
I20260923 21:01:08.411810 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.368083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43235 (local address 127.0.97.61:33323)
0923 21:01:08.374902 (+ 6819us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.374913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.375425 (+ 512us) server_negotiation.cc:407] Connection header received
0923 21:01:08.377030 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.377041 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.377158 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.377261 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.380697 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.381625 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.384465 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.385247 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.385389 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.392489 (+ 7100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.392502 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.392512 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.392581 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.398056 (+ 5475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.409325 (+ 11269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.409334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.409340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.409555 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.410057 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.410063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.410066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.410446 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.410619 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.411154 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.411394 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6631,"thread_start_us":6483,"threads_started":1}
I20260923 21:01:08.412569 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.353979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.97.3:60739)
0923 21:01:08.363844 (+ 9865us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.363932 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.376583 (+ 12651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.379388 (+ 2805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.379397 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.380166 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:08.380761 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.380780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.383117 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.383126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.385652 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.385679 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.388266 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.388281 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.388637 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.389572 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:08.389599 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:08.393383 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:08.402615 (+ 9232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:08.402622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:08.402635 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:08.403251 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:08.406085 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:08.406090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:08.406094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:08.406318 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:08.408051 (+ 1733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:08.408057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:08.408236 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.408376 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.408705 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9331,"mutex_wait_us":52,"thread_start_us":12178,"threads_started":1}
I20260923 21:01:08.413014 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.374132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.97.3:43235)
0923 21:01:08.375260 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.375292 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.376608 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.377688 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.377700 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.379140 (+ 1440us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:08.380450 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.380475 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.382547 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.382553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.384282 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.384301 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.385729 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.385736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.386099 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.388096 (+ 1997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:08.388150 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:08.392094 (+ 3944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:08.408667 (+ 16573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:08.408672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:08.408676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:08.409162 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:08.409718 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:08.409722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:08.409725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:08.409899 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:08.410697 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:08.410701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:08.410844 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.412177 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.412507 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":86,"thread_start_us":1271,"threads_started":1}
I20260923 21:01:08.413129 6549 ts_manager.cc:194] Re-registered known tserver with Master: ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747)
I20260923 21:01:08.414439 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.353012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60739 (local address 127.0.97.60:46465)
0923 21:01:08.378643 (+ 25631us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.378654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.378684 (+ 30us) server_negotiation.cc:407] Connection header received
0923 21:01:08.378834 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.378843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.378955 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.379121 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.381024 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.382784 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.385933 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.386970 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.387151 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.393646 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.393655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.393661 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.393708 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.402396 (+ 8688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.405547 (+ 3151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.405556 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.405560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.405770 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.407411 (+ 1641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.407417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.407421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.407799 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.407949 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.413846 (+ 5897us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.414095 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25458,"thread_start_us":25311,"threads_started":1}
I20260923 21:01:08.414913 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39393
I20260923 21:01:08.415473 7807 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46465
I20260923 21:01:08.415625 7807 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:08.416692 7809 heartbeater.cc:347] Connected to a master server at 127.0.97.61:33323
I20260923 21:01:08.416788 7807 heartbeater.cc:511] Master 127.0.97.60:46465 requested a full tablet report, sending...
I20260923 21:01:08.416855 7809 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:08.417492 7809 heartbeater.cc:511] Master 127.0.97.61:33323 requested a full tablet report, sending...
I20260923 21:01:08.418772 7339 ts_manager.cc:194] Registered new tserver with Master: ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747)
I20260923 21:01:08.419677 7149 ts_manager.cc:194] Registered new tserver with Master: ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747)
I20260923 21:01:08.447546 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.395712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.0.97.61:33323)
0923 21:01:08.412128 (+ 16416us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.412134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.427290 (+ 15156us) server_negotiation.cc:407] Connection header received
0923 21:01:08.427624 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.427633 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.427729 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.427819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.431324 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.433101 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.435350 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.436105 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.436216 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.440532 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.440538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.440545 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.440587 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.444554 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.445259 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.445265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.445268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.445402 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.445877 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.445882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.445886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.446195 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.446498 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.446908 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.447162 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16300,"thread_start_us":26655,"threads_started":1}
I20260923 21:01:08.447757 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.396398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.0.97.62:39963)
0923 21:01:08.408593 (+ 12195us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.408599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.427109 (+ 18510us) server_negotiation.cc:407] Connection header received
0923 21:01:08.427326 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.427333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.427446 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.427583 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.430853 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.431934 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.434437 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.435032 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.435212 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.439648 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.439653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.439656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.439691 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.443259 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.445230 (+ 1971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.445237 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.445241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.445404 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.445878 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.445884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.445890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.446386 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.446661 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.447050 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.447331 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12075,"thread_start_us":29769,"threads_started":1}
I20260923 21:01:08.449827 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.427330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.0.97.60:46465)
0923 21:01:08.427535 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.427541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.429571 (+ 2030us) server_negotiation.cc:407] Connection header received
0923 21:01:08.429803 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.429810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.429930 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.430040 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.433484 (+ 3444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.434759 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.436948 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.437690 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.437857 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.441874 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.441881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.441886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.441934 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.446214 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.447283 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.447290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.447294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.447476 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.447997 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.448003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.448007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.448368 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.448646 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.449241 (+ 595us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.449438 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 21:01:08.453258 6549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:01:08.461336 6549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:08.500295 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.486715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.0.97.2:41743)
0923 21:01:08.487304 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.487314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.490151 (+ 2837us) server_negotiation.cc:407] Connection header received
0923 21:01:08.490494 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.490505 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.490807 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.491030 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:08.492075 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.493570 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.497083 (+ 3513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.498727 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.498842 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.499085 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.499785 (+ 700us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.499992 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":358,"threads_started":1}
I20260923 21:01:08.500727 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.485674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.0.97.3:34747)
0923 21:01:08.486409 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.486420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.490046 (+ 3626us) server_negotiation.cc:407] Connection header received
0923 21:01:08.490525 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.490531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.490751 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.490983 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:08.492289 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.493851 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.497443 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.499132 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.499246 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.499483 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.500165 (+ 682us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.500370 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":345,"threads_started":1}
I20260923 21:01:08.500809 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.490562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.0.97.1:38547)
0923 21:01:08.491353 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.491368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.492235 (+ 867us) server_negotiation.cc:407] Connection header received
0923 21:01:08.492355 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.492363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.492643 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.492853 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:08.493465 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.495029 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.497845 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.499390 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.499503 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.499725 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.500344 (+ 619us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.500512 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":400,"threads_started":1}
I20260923 21:01:08.501266 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.486296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41743 (local address 127.0.0.1:48016)
0923 21:01:08.490035 (+ 3739us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.490053 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.490267 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.491271 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.491277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.491303 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:08.491864 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.491879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.493751 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.493756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.496874 (+ 3118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.496890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.499296 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.499302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.499529 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.499581 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.500617 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.500785 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3610,"mutex_wait_us":72,"spinlock_wait_cycles":1408,"thread_start_us":2663,"threads_started":1}
I20260923 21:01:08.501710 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.491301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38547 (local address 127.0.0.1:50128)
0923 21:01:08.492069 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.492089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.492222 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.492864 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.492869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.492896 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:08.493265 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.493279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.495243 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.495248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.497647 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.497663 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.499945 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.499951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.500144 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.500175 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.501044 (+ 869us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.501234 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":28,"thread_start_us":366,"threads_started":1}
I20260923 21:01:08.502735 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.485174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34747 (local address 127.0.0.1:45912)
0923 21:01:08.489840 (+ 4666us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.489867 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.490264 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.491505 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.491513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.491567 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:08.492040 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.492058 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.494094 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.494103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.497224 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.497245 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.499726 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.499732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.499940 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.499971 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.501021 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.501234 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4487,"mutex_wait_us":87,"spinlock_wait_cycles":2944,"thread_start_us":4538,"threads_started":1}
I20260923 21:01:08.505463 7453 tablet_service.cc:1467] Processing CreateTablet for tablet f53eb0a0173840f2a483c56b76a828f0 (DEFAULT_TABLE table=test-table [id=eda54a19807441c8a504b4b903917693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:08.505601 7601 tablet_service.cc:1467] Processing CreateTablet for tablet f53eb0a0173840f2a483c56b76a828f0 (DEFAULT_TABLE table=test-table [id=eda54a19807441c8a504b4b903917693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:08.506771 7742 tablet_service.cc:1467] Processing CreateTablet for tablet f53eb0a0173840f2a483c56b76a828f0 (DEFAULT_TABLE table=test-table [id=eda54a19807441c8a504b4b903917693]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:08.507402 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53eb0a0173840f2a483c56b76a828f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:08.507597 7453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53eb0a0173840f2a483c56b76a828f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:08.508236 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53eb0a0173840f2a483c56b76a828f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:08.523450 7843 tablet_bootstrap.cc:492] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Bootstrap starting.
I20260923 21:01:08.526698 7843 tablet_bootstrap.cc:654] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:08.527773 7843 log.cc:824] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:08.528813 7844 tablet_bootstrap.cc:492] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Bootstrap starting.
I20260923 21:01:08.531733 7846 tablet_bootstrap.cc:492] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Bootstrap starting.
I20260923 21:01:08.533304 7843 tablet_bootstrap.cc:492] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: No bootstrap required, opened a new log
I20260923 21:01:08.533771 7843 ts_tablet_manager.cc:1397] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260923 21:01:08.534088 7844 tablet_bootstrap.cc:654] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:08.535454 7846 tablet_bootstrap.cc:654] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:08.535452 7844 log.cc:824] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:08.536684 7846 log.cc:824] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:08.539073 7844 tablet_bootstrap.cc:492] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: No bootstrap required, opened a new log
I20260923 21:01:08.539377 7844 ts_tablet_manager.cc:1397] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 21:01:08.540879 7846 tablet_bootstrap.cc:492] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: No bootstrap required, opened a new log
I20260923 21:01:08.541154 7846 ts_tablet_manager.cc:1397] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 21:01:08.544186 7843 raft_consensus.cc:348] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.544688 7843 raft_consensus.cc:374] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:08.544786 7843 raft_consensus.cc:729] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdba6177147a47e693ad2d152127f63a, State: Initialized, Role: FOLLOWER
I20260923 21:01:08.545957 7843 consensus_queue.cc:260] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.547974 7844 raft_consensus.cc:348] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.548393 7844 raft_consensus.cc:374] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:08.548534 7844 raft_consensus.cc:729] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf0c67eda0742cda4369580b7ba83fa, State: Initialized, Role: FOLLOWER
I20260923 21:01:08.549341 7844 consensus_queue.cc:260] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.549921 7843 ts_tablet_manager.cc:1428] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260923 21:01:08.552814 7846 raft_consensus.cc:348] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.553333 7846 raft_consensus.cc:374] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:08.553454 7846 raft_consensus.cc:729] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5abb06795f44babcdb3f3f4040a426, State: Initialized, Role: FOLLOWER
I20260923 21:01:08.554283 7846 consensus_queue.cc:260] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.558130 7844 ts_tablet_manager.cc:1428] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Time spent starting tablet: real 0.019s user 0.010s sys 0.005s
I20260923 21:01:08.558514 7808 heartbeater.cc:503] Master 127.0.97.62:39963 was elected leader, sending a full tablet report...
I20260923 21:01:08.563098 7667 heartbeater.cc:503] Master 127.0.97.62:39963 was elected leader, sending a full tablet report...
I20260923 21:01:08.563366 7846 ts_tablet_manager.cc:1428] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Time spent starting tablet: real 0.022s user 0.018s sys 0.005s
I20260923 21:01:08.609329 7849 raft_consensus.cc:482] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:08.609686 7849 raft_consensus.cc:504] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.611651 7849 leader_election.cc:288] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747), 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:08.619536 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.612234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37407 (local address 127.0.97.2:41743)
0923 21:01:08.612475 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.612481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.613212 (+ 731us) server_negotiation.cc:407] Connection header received
0923 21:01:08.613375 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.613379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.613451 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.613610 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:08.614276 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.615152 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.617471 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.618433 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.618558 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.618637 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.619128 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.619276 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 21:01:08.619540 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.612039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41743 (local address 127.0.97.1:37407)
0923 21:01:08.613082 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.613104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.613267 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.613634 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.613640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.613664 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:08.614093 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.614108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.615301 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.615307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.617266 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.617284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.618659 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.618666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.618845 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.618896 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.619023 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.619178 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":96,"thread_start_us":445,"threads_started":1}
I20260923 21:01:08.620075 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.612294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34383 (local address 127.0.97.3:34747)
0923 21:01:08.612489 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.612497 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.613104 (+ 607us) server_negotiation.cc:407] Connection header received
0923 21:01:08.613314 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.613322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.613409 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.613634 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:08.614321 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.615158 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.617365 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.618968 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.619089 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.619185 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.619701 (+ 516us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.619848 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 21:01:08.620796 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.612039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34747 (local address 127.0.97.1:34383)
0923 21:01:08.612969 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:08.612989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:08.613147 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:08.613646 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:08.613651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:08.613676 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:08.614146 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.614164 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.615323 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.615328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.617185 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:08.617198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.619213 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.619220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:08.619405 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.619439 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 21:01:08.620264 (+ 825us) client_negotiation.cc:240] Negotiation successful
0923 21:01:08.620425 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":18,"thread_start_us":619,"threads_started":1}
I20260923 21:01:08.620798 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53eb0a0173840f2a483c56b76a828f0" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5abb06795f44babcdb3f3f4040a426" is_pre_election: true
I20260923 21:01:08.621407 7621 raft_consensus.cc:2454] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 0.
I20260923 21:01:08.622097 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53eb0a0173840f2a483c56b76a828f0" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf0c67eda0742cda4369580b7ba83fa" is_pre_election: true
I20260923 21:01:08.622224 7407 leader_election.cc:302] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5abb06795f44babcdb3f3f4040a426, cdba6177147a47e693ad2d152127f63a; no voters:
I20260923 21:01:08.622648 7762 raft_consensus.cc:2454] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 0.
I20260923 21:01:08.622740 7849 raft_consensus.cc:2781] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:08.622915 7849 raft_consensus.cc:482] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:08.623032 7849 raft_consensus.cc:3037] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Advancing to term 1
W20260923 21:01:08.626060 7810 tablet.cc:2369] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:08.626150 7849 raft_consensus.cc:504] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.627004 7849 leader_election.cc:288] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 election: Requested vote from peers ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747), 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:08.627542 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53eb0a0173840f2a483c56b76a828f0" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5abb06795f44babcdb3f3f4040a426"
I20260923 21:01:08.627540 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53eb0a0173840f2a483c56b76a828f0" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf0c67eda0742cda4369580b7ba83fa"
I20260923 21:01:08.627768 7762 raft_consensus.cc:3037] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:08.627768 7621 raft_consensus.cc:3037] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:08.631110 7762 raft_consensus.cc:2454] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 1.
I20260923 21:01:08.631601 7408 leader_election.cc:302] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdba6177147a47e693ad2d152127f63a, ddf0c67eda0742cda4369580b7ba83fa; no voters:
I20260923 21:01:08.631664 7621 raft_consensus.cc:2454] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 1.
I20260923 21:01:08.631999 7849 raft_consensus.cc:2781] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:08.632507 7849 raft_consensus.cc:686] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 1 LEADER]: Becoming Leader. State: Replica: cdba6177147a47e693ad2d152127f63a, State: Running, Role: LEADER
I20260923 21:01:08.633349 7849 consensus_queue.cc:237] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:08.639281 6548 catalog_manager.cc:5495] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a reported cstate change: term changed from 0 to 1, leader changed from <none> to cdba6177147a47e693ad2d152127f63a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "cdba6177147a47e693ad2d152127f63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } health_report { overall_health: UNKNOWN } } }
I20260923 21:01:08.673086 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.667333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.0.97.1:38547)
0923 21:01:08.667566 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.667573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.667748 (+ 175us) server_negotiation.cc:407] Connection header received
0923 21:01:08.668021 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.668025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.668121 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.668291 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:08.668655 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.669424 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.670827 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.671241 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.671382 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.671875 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:08.672407 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:08.672576 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.672715 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.672799 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260923 21:01:08.682251 7669 tablet.cc:2369] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:08.683171 7621 raft_consensus.cc:1261] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Refusing update from remote peer cdba6177147a47e693ad2d152127f63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:08.683159 7762 raft_consensus.cc:1261] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Refusing update from remote peer cdba6177147a47e693ad2d152127f63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:08.684226 7854 consensus_queue.cc:1035] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:08.685204 7854 consensus_queue.cc:1035] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:08.709440 7858 mvcc.cc:205] Tried to move back new op lower bound from 7332648012514377728 to 7332648012326072320. Current Snapshot: MvccSnapshot[applied={T|T < 7332648012514377728}]
I20260923 21:01:08.712146 7857 mvcc.cc:205] Tried to move back new op lower bound from 7332648012514377728 to 7332648012326072320. Current Snapshot: MvccSnapshot[applied={T|T < 7332648012514377728 or (T in {7332648012514377728})}]
I20260923 21:01:08.736274 6548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:08.736588 6548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:08.744550 6548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 709239c2e0d0464882bf6056e5b9c0b3: Sending DeleteTablet for 3 replicas of tablet f53eb0a0173840f2a483c56b76a828f0
I20260923 21:01:08.745786 7742 tablet_service.cc:1514] Processing DeleteTablet for tablet f53eb0a0173840f2a483c56b76a828f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45912
I20260923 21:01:08.745970 7453 tablet_service.cc:1514] Processing DeleteTablet for tablet f53eb0a0173840f2a483c56b76a828f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50128
I20260923 21:01:08.746061 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet f53eb0a0173840f2a483c56b76a828f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48016
I20260923 21:01:08.761755 7869 tablet_replica.cc:330] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: stopping tablet replica
I20260923 21:01:08.763284 7867 tablet_replica.cc:330] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: stopping tablet replica
I20260923 21:01:08.763592 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 6517
I20260923 21:01:08.763804 7867 raft_consensus.cc:2229] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:08.763899 7868 tablet_replica.cc:330] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: stopping tablet replica
I20260923 21:01:08.764499 7868 raft_consensus.cc:2229] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:01:08.764953 6579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:08.765209 7868 raft_consensus.cc:2258] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:08.765547 7869 raft_consensus.cc:2229] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:08.766127 7869 raft_consensus.cc:2258] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:08.767328 7867 raft_consensus.cc:2258] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:08.769982 7868 ts_tablet_manager.cc:1905] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:08.770059 7867 ts_tablet_manager.cc:1905] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:08.772370 7869 ts_tablet_manager.cc:1905] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:08.777391 7867 ts_tablet_manager.cc:1918] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:08.777618 7867 log.cc:1196] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/f53eb0a0173840f2a483c56b76a828f0
I20260923 21:01:08.778286 7867 ts_tablet_manager.cc:1939] T f53eb0a0173840f2a483c56b76a828f0 P ddf0c67eda0742cda4369580b7ba83fa: Deleting consensus metadata
I20260923 21:01:08.785284 7869 ts_tablet_manager.cc:1918] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:08.785537 7869 log.cc:1196] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/f53eb0a0173840f2a483c56b76a828f0
I20260923 21:01:08.786125 7869 ts_tablet_manager.cc:1939] T f53eb0a0173840f2a483c56b76a828f0 P 5e5abb06795f44babcdb3f3f4040a426: Deleting consensus metadata
I20260923 21:01:08.788631 7868 ts_tablet_manager.cc:1918] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:08.788844 7868 log.cc:1196] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/f53eb0a0173840f2a483c56b76a828f0
I20260923 21:01:08.789512 7868 ts_tablet_manager.cc:1939] T f53eb0a0173840f2a483c56b76a828f0 P cdba6177147a47e693ad2d152127f63a: Deleting consensus metadata
I20260923 21:01:08.794260 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.749737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.0.97.61:33323)
0923 21:01:08.749925 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.749931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.762598 (+ 12667us) server_negotiation.cc:407] Connection header received
0923 21:01:08.763184 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.763191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.763282 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.763397 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.767086 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.768369 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.775961 (+ 7592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.776692 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.776824 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.786690 (+ 9866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.786698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.786704 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.786758 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.790673 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.791802 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.791810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.791814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.792020 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.792520 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.792525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.792528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.792875 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.793130 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.793571 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.793831 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:01:08.813911 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.762896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.0.97.60:46465)
0923 21:01:08.763080 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.763086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.794418 (+ 31332us) server_negotiation.cc:407] Connection header received
0923 21:01:08.794626 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.794633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.794717 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.794824 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.797855 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.799228 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.801331 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.802070 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.802234 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.805852 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.805859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.805865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.805914 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.810009 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.811057 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.811065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.811069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.811228 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.811723 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.811728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.811732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.812119 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.812383 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.813246 (+ 863us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.813529 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:01:08.908703 6534 catalog_manager.cc:4841] TS 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743): tablet f53eb0a0173840f2a483c56b76a828f0 (table test-table [id=eda54a19807441c8a504b4b903917693]) successfully deleted
I20260923 21:01:08.909687 6535 catalog_manager.cc:4841] TS ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747): tablet f53eb0a0173840f2a483c56b76a828f0 (table test-table [id=eda54a19807441c8a504b4b903917693]) successfully deleted
W20260923 21:01:08.909915 6535 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:34747, user_credentials={real_user=slave}} blocked reactor thread for 129144us
I20260923 21:01:08.910315 6535 catalog_manager.cc:4841] TS cdba6177147a47e693ad2d152127f63a (127.0.97.1:38547): tablet f53eb0a0173840f2a483c56b76a828f0 (table test-table [id=eda54a19807441c8a504b4b903917693]) successfully deleted
I20260923 21:01:08.910636 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 6517
I20260923 21:01:08.960638 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.941793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.0.97.61:33323)
0923 21:01:08.942000 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.942008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.943290 (+ 1282us) server_negotiation.cc:407] Connection header received
0923 21:01:08.943592 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.943597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.943689 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.943762 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.946660 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.947446 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.949437 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.950079 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.950223 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.953488 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.953494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.953497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.953545 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.957278 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.958274 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.958280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.958284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.958483 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.958959 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.958964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.958967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.959393 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.959652 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.960009 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.960250 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:01:08.961668 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:08.943304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.0.97.60:46465)
0923 21:01:08.943545 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:08.943552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:08.943576 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:01:08.943713 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:08.943719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:08.943793 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:08.943900 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:08.946642 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.947771 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.949828 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:08.950327 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:08.950749 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:08.954525 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:08.954533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:08.954541 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:08.954588 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:08.958071 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.958862 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.958869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.958872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.959022 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:08.959482 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:08.959486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:08.959489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:08.959859 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:08.960096 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:08.961084 (+ 988us) server_negotiation.cc:299] Negotiation successful
0923 21:01:08.961356 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 21:01:09.724964 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:09.724131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:33685)
0923 21:01:09.724652 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:09.724777 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":279,"threads_started":1}
I20260923 21:01:09.725203 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:09.724261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:59763)
0923 21:01:09.724880 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:09.725008 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":412,"threads_started":1}
W20260923 21:01:09.725455 7808 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
W20260923 21:01:09.725549 7519 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:09.733400 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:09.732337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:45061)
0923 21:01:09.732913 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:09.733081 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":292,"threads_started":1}
W20260923 21:01:09.733937 7667 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:10.244366 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 709239c2e0d0464882bf6056e5b9c0b3)
I20260923 21:01:10.244750 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:10.245633 7882 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 709239c2e0d0464882bf6056e5b9c0b3)
I20260923 21:01:10.245910 7882 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:10.246794 7881 leader_election.cc:288] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963), f90374f37df945d1805e836834117852 (127.0.97.61:33323)
I20260923 21:01:10.256202 7882 leader_election.cc:288] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963), a1a59587862b4f15be79496c93feb622 (127.0.97.60:46465)
I20260923 21:01:10.263188 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.246991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48484)
0923 21:01:10.262480 (+ 15489us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.262742 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15261,"thread_start_us":15625,"threads_started":1}
W20260923 21:01:10.263546 7324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:10.264245 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.247684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48494)
0923 21:01:10.262953 (+ 15269us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.263203 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15101,"thread_start_us":15728,"threads_started":1}
W20260923 21:01:10.264703 7134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:10.267248 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.265906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48508)
0923 21:01:10.266738 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.266955 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":673}
W20260923 21:01:10.267917 7324 leader_election.cc:334] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:10.271395 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.269753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48518)
0923 21:01:10.270923 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.271148 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":992}
W20260923 21:01:10.271642 7134 leader_election.cc:334] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:10.276805 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.257244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46465 (local address 127.0.0.1:35338)
0923 21:01:10.265478 (+ 8234us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.265499 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:10.265655 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:10.270861 (+ 5206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:10.270868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:10.270921 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:10.271274 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:10.271294 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.273232 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.273237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:10.274721 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:10.274733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.276045 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.276051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:10.276236 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:10.276274 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 21:01:10.276386 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 21:01:10.276539 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8125,"thread_start_us":13316,"threads_started":1}
I20260923 21:01:10.277077 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.265253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.0.97.60:46465)
0923 21:01:10.270431 (+ 5178us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:10.270441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:10.270474 (+ 33us) server_negotiation.cc:407] Connection header received
0923 21:01:10.270569 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:10.270578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:10.270664 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:10.271451 (+ 787us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:10.272185 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.273075 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.274892 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.275849 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.275971 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:10.276053 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:10.276538 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 21:01:10.276687 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4929,"thread_start_us":4793,"threads_started":1}
I20260923 21:01:10.277388 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.247300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.97.61:33323)
0923 21:01:10.271309 (+ 24009us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:10.271315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:10.271334 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:01:10.271391 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:10.271397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:10.271466 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:10.271562 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:10.272383 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.273327 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.275287 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.276371 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.276477 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:10.276588 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:10.276929 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 21:01:10.277064 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23827,"thread_start_us":23593,"threads_started":1}
I20260923 21:01:10.277446 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.247186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33323 (local address 127.0.0.1:53990)
0923 21:01:10.263504 (+ 16318us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.263532 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:10.263743 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:10.271877 (+ 8134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:10.271882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:10.271915 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:10.272173 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:10.272184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.273519 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.273523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:10.275129 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:10.275143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:10.276552 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:10.276558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:10.276748 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:10.276781 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 21:01:10.276930 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 21:01:10.277080 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16194,"thread_start_us":20271,"threads_started":1}
I20260923 21:01:10.278105 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1a59587862b4f15be79496c93feb622" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f90374f37df945d1805e836834117852" is_pre_election: true
I20260923 21:01:10.278117 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f90374f37df945d1805e836834117852" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a1a59587862b4f15be79496c93feb622" is_pre_election: true
I20260923 21:01:10.278469 7350 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f90374f37df945d1805e836834117852 in term 1.
I20260923 21:01:10.278478 7159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1a59587862b4f15be79496c93feb622 in term 1.
I20260923 21:01:10.278976 7133 leader_election.cc:302] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a1a59587862b4f15be79496c93feb622, f90374f37df945d1805e836834117852; no voters: 709239c2e0d0464882bf6056e5b9c0b3
I20260923 21:01:10.279007 7323 leader_election.cc:302] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a1a59587862b4f15be79496c93feb622, f90374f37df945d1805e836834117852; no voters: 709239c2e0d0464882bf6056e5b9c0b3
I20260923 21:01:10.279331 7882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 21:01:10.279414 7881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 21:01:10.279459 7882 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 709239c2e0d0464882bf6056e5b9c0b3)
I20260923 21:01:10.279549 7882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 1 FOLLOWER]: Advancing to term 2
I20260923 21:01:10.279668 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 709239c2e0d0464882bf6056e5b9c0b3)
I20260923 21:01:10.279803 7881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 1 FOLLOWER]: Advancing to term 2
I20260923 21:01:10.282111 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:10.282102 7882 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:10.282835 7881 leader_election.cc:288] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [CANDIDATE]: Term 2 election: Requested vote from peers 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963), f90374f37df945d1805e836834117852 (127.0.97.61:33323)
I20260923 21:01:10.282977 7882 leader_election.cc:288] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 2 election: Requested vote from peers 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963), a1a59587862b4f15be79496c93feb622 (127.0.97.60:46465)
I20260923 21:01:10.283630 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.283066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48526)
0923 21:01:10.283349 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.283445 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260923 21:01:10.283607 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1a59587862b4f15be79496c93feb622" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f90374f37df945d1805e836834117852"
I20260923 21:01:10.283605 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f90374f37df945d1805e836834117852" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a1a59587862b4f15be79496c93feb622"
I20260923 21:01:10.283874 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.283342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48534)
0923 21:01:10.283580 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.283676 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260923 21:01:10.283955 7159 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a1a59587862b4f15be79496c93feb622 in current term 2: Already voted for candidate f90374f37df945d1805e836834117852 in this term.
I20260923 21:01:10.283987 7350 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f90374f37df945d1805e836834117852 in current term 2: Already voted for candidate a1a59587862b4f15be79496c93feb622 in this term.
I20260923 21:01:10.284816 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.284435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48546)
0923 21:01:10.284633 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.284715 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 21:01:10.284932 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.284449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48548)
0923 21:01:10.284690 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.284766 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260923 21:01:10.284991 7324 leader_election.cc:334] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:10.285133 7324 leader_election.cc:302] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1a59587862b4f15be79496c93feb622; no voters: 709239c2e0d0464882bf6056e5b9c0b3, f90374f37df945d1805e836834117852
W20260923 21:01:10.285166 7134 leader_election.cc:334] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:10.285322 7134 leader_election.cc:302] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f90374f37df945d1805e836834117852; no voters: 709239c2e0d0464882bf6056e5b9c0b3, a1a59587862b4f15be79496c93feb622
I20260923 21:01:10.285450 7881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 21:01:10.285622 7882 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 21:01:10.727185 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.726242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:40171)
0923 21:01:10.726787 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.726937 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":542,"threads_started":1}
I20260923 21:01:10.727404 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.726655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:55311)
0923 21:01:10.727103 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.727191 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":314,"threads_started":1}
I20260923 21:01:10.739213 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:10.734857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:54723)
0923 21:01:10.738835 (+ 3978us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:10.739017 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3831,"thread_start_us":3699,"threads_started":1}
I20260923 21:01:11.729398 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:11.728343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:45081)
0923 21:01:11.729014 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:11.729162 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":370,"threads_started":1}
I20260923 21:01:11.729785 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:11.728338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:36901)
0923 21:01:11.729364 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:11.729551 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":799,"threads_started":1}
I20260923 21:01:11.741130 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:11.740375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:33645)
0923 21:01:11.740845 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:11.740934 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":258,"threads_started":1}
I20260923 21:01:12.119650 7897 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:12.119946 7897 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:12.120656 7897 leader_election.cc:288] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963), a1a59587862b4f15be79496c93feb622 (127.0.97.60:46465)
I20260923 21:01:12.121254 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f90374f37df945d1805e836834117852" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a1a59587862b4f15be79496c93feb622" is_pre_election: true
I20260923 21:01:12.121667 7350 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f90374f37df945d1805e836834117852 in term 2.
I20260923 21:01:12.121999 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.120935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48844)
0923 21:01:12.121622 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.121738 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":364,"threads_started":1}
I20260923 21:01:12.122246 7133 leader_election.cc:302] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1a59587862b4f15be79496c93feb622, f90374f37df945d1805e836834117852; no voters:
I20260923 21:01:12.122750 7897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 21:01:12.122946 7897 raft_consensus.cc:482] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:12.123071 7897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 2 FOLLOWER]: Advancing to term 3
I20260923 21:01:12.124011 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.123423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48848)
0923 21:01:12.123767 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.123862 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242}
W20260923 21:01:12.124213 7134 leader_election.cc:334] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:12.125717 7897 raft_consensus.cc:504] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:12.126405 7897 leader_election.cc:288] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 3 election: Requested vote from peers 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963), a1a59587862b4f15be79496c93feb622 (127.0.97.60:46465)
I20260923 21:01:12.127071 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.126613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48856)
0923 21:01:12.126823 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.126926 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 21:01:12.126998 7350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f90374f37df945d1805e836834117852" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a1a59587862b4f15be79496c93feb622"
I20260923 21:01:12.127233 7350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 2 FOLLOWER]: Advancing to term 3
I20260923 21:01:12.128089 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.127747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48864)
0923 21:01:12.127901 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.127977 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 21:01:12.128294 7134 leader_election.cc:334] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
I20260923 21:01:12.130053 7350 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f90374f37df945d1805e836834117852 in term 3.
I20260923 21:01:12.130668 7133 leader_election.cc:302] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a1a59587862b4f15be79496c93feb622, f90374f37df945d1805e836834117852; no voters: 709239c2e0d0464882bf6056e5b9c0b3
I20260923 21:01:12.131065 7897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 21:01:12.131659 7897 raft_consensus.cc:686] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [term 3 LEADER]: Becoming Leader. State: Replica: f90374f37df945d1805e836834117852, State: Running, Role: LEADER
I20260923 21:01:12.132164 7897 consensus_queue.cc:237] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } }
I20260923 21:01:12.135088 7900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90374f37df945d1805e836834117852. Latest consensus state: current_term: 3 leader_uuid: "f90374f37df945d1805e836834117852" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:12.135401 7900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:12.136211 7902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:12.140825 7902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:12.141839 7902 catalog_manager.cc:1259] Loaded cluster ID: 6b5f8a05b56b494e9540ffb2aa37be8d
I20260923 21:01:12.141968 7902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:12.143421 7902 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:12.144345 7902 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Loaded TSK: 0
I20260923 21:01:12.145315 7902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:01:12.191757 7149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:01:12.199468 7149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:12.204305 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.203804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48892)
0923 21:01:12.204044 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.204127 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 21:01:12.204490 7350 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [term 3 FOLLOWER]: Refusing update from remote peer f90374f37df945d1805e836834117852: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 21:01:12.205423 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.205014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:48908)
0923 21:01:12.205222 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.205300 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 21:01:12.205269 7900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260923 21:01:12.205643 7134 consensus_peers.cc:487] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 -> Peer 709239c2e0d0464882bf6056e5b9c0b3 (127.0.97.62:39963): Couldn't send request to peer 709239c2e0d0464882bf6056e5b9c0b3. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 21:01:12.208556 7904 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f90374f37df945d1805e836834117852. Latest consensus state: current_term: 3 leader_uuid: "f90374f37df945d1805e836834117852" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:12.208909 7904 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:01:12.214659 7904 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f90374f37df945d1805e836834117852" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:12.215067 7904 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1a59587862b4f15be79496c93feb622 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:01:12.218545 7897 sys_catalog.cc:455] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f90374f37df945d1805e836834117852" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "709239c2e0d0464882bf6056e5b9c0b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39963 } } peers { permanent_uuid: "f90374f37df945d1805e836834117852" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33323 } attrs { promote: false } } peers { permanent_uuid: "a1a59587862b4f15be79496c93feb622" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46465 } attrs { promote: false } } }
I20260923 21:01:12.218904 7897 sys_catalog.cc:458] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:12.234184 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.226274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.0.97.1:38547)
0923 21:01:12.227687 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:12.227694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:12.227715 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:01:12.227808 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:12.227813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:12.227902 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:12.228026 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:12.228832 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.229901 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.232011 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.233061 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.233221 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.233339 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:12.233716 (+ 377us) server_negotiation.cc:299] Negotiation successful
0923 21:01:12.233865 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1153,"threads_started":1}
I20260923 21:01:12.234282 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.226156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38547 (local address 127.0.0.1:50150)
0923 21:01:12.226379 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.226403 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:12.226536 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:12.228114 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:12.228124 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:12.228171 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:12.228642 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:12.228659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.230071 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.230076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:12.231809 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:12.231829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.233310 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.233316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:12.233545 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.233571 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 21:01:12.233728 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 21:01:12.233919 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 21:01:12.239854 7453 tablet_service.cc:1467] Processing CreateTablet for tablet 5be8d930682d4d20aa4fca5ae93377e4 (DEFAULT_TABLE table=test-table [id=aabd2abb1a1343d6845063067866e799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:12.240036 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.226662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34747 (local address 127.0.0.1:52906)
0923 21:01:12.234543 (+ 7881us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.234567 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:12.234710 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:12.235162 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:12.235169 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:12.235197 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:12.235347 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:12.235354 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.236359 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.236363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:12.237592 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:12.237604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.239202 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.239207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:12.239435 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.239473 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 21:01:12.239573 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 21:01:12.239716 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7764,"thread_start_us":11941,"threads_started":1}
I20260923 21:01:12.240281 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.226918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.0.97.3:34747)
0923 21:01:12.227823 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:12.227828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:12.234653 (+ 6825us) server_negotiation.cc:407] Connection header received
0923 21:01:12.234853 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:12.234858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:12.234976 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:12.235096 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:12.235485 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.236221 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.237768 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.238931 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.239202 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.239392 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:12.239622 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 21:01:12.239774 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":955,"threads_started":1}
I20260923 21:01:12.240444 7453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be8d930682d4d20aa4fca5ae93377e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:12.241490 7742 tablet_service.cc:1467] Processing CreateTablet for tablet 5be8d930682d4d20aa4fca5ae93377e4 (DEFAULT_TABLE table=test-table [id=aabd2abb1a1343d6845063067866e799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:12.241940 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be8d930682d4d20aa4fca5ae93377e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:12.248096 7916 tablet_bootstrap.cc:492] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Bootstrap starting.
I20260923 21:01:12.249306 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.227584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41743 (local address 127.0.0.1:49312)
0923 21:01:12.238688 (+ 11104us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.238716 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:12.240126 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:12.240858 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:12.240861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:12.240881 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:12.241210 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:12.241222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.242939 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.242947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:12.245339 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:12.245359 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.247981 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.247991 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:12.248385 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.248436 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 21:01:12.248556 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 21:01:12.248777 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10890,"thread_start_us":11253,"threads_started":1}
I20260923 21:01:12.249787 7916 tablet_bootstrap.cc:654] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:12.250375 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.228099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49312 (local address 127.0.97.2:41743)
0923 21:01:12.228631 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:12.228636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:12.238827 (+ 10191us) server_negotiation.cc:407] Connection header received
0923 21:01:12.240330 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:12.240336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:12.240423 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:12.240589 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:12.241412 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.242734 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.245570 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.247418 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.247597 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.247792 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:12.249678 (+ 1886us) server_negotiation.cc:299] Negotiation successful
0923 21:01:12.249930 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":298,"threads_started":1}
I20260923 21:01:12.251247 7915 tablet_bootstrap.cc:492] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: Bootstrap starting.
I20260923 21:01:12.251215 7601 tablet_service.cc:1467] Processing CreateTablet for tablet 5be8d930682d4d20aa4fca5ae93377e4 (DEFAULT_TABLE table=test-table [id=aabd2abb1a1343d6845063067866e799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:12.251673 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be8d930682d4d20aa4fca5ae93377e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:12.254030 7915 tablet_bootstrap.cc:654] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:12.257041 7919 tablet_bootstrap.cc:492] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: Bootstrap starting.
I20260923 21:01:12.257126 7915 tablet_bootstrap.cc:492] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: No bootstrap required, opened a new log
I20260923 21:01:12.257345 7915 ts_tablet_manager.cc:1397] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 21:01:12.258630 7915 raft_consensus.cc:348] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.258930 7915 raft_consensus.cc:374] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:12.259030 7915 raft_consensus.cc:729] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf0c67eda0742cda4369580b7ba83fa, State: Initialized, Role: FOLLOWER
I20260923 21:01:12.259150 7919 tablet_bootstrap.cc:654] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:12.259286 7915 consensus_queue.cc:260] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.260167 7916 tablet_bootstrap.cc:492] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: No bootstrap required, opened a new log
I20260923 21:01:12.260186 7915 ts_tablet_manager.cc:1428] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 21:01:12.260349 7916 ts_tablet_manager.cc:1397] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260923 21:01:12.261317 7916 raft_consensus.cc:348] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.261583 7916 raft_consensus.cc:374] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:12.261685 7916 raft_consensus.cc:729] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdba6177147a47e693ad2d152127f63a, State: Initialized, Role: FOLLOWER
I20260923 21:01:12.261976 7916 consensus_queue.cc:260] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.266757 7809 heartbeater.cc:503] Master 127.0.97.61:33323 was elected leader, sending a full tablet report...
I20260923 21:01:12.271507 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.260801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:53301)
0923 21:01:12.270901 (+ 10100us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.271111 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9896,"thread_start_us":9769,"threads_started":1}
I20260923 21:01:12.271652 7916 ts_tablet_manager.cc:1428] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
I20260923 21:01:12.273372 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.272464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:45869)
0923 21:01:12.273113 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.273202 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539}
I20260923 21:01:12.273496 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.272255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:51773)
0923 21:01:12.272945 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.273157 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":416,"threads_started":1}
I20260923 21:01:12.274976 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.274414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:38489)
0923 21:01:12.274697 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.274799 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260923 21:01:12.275830 7520 heartbeater.cc:503] Master 127.0.97.61:33323 was elected leader, sending a full tablet report...
I20260923 21:01:12.280083 7919 tablet_bootstrap.cc:492] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: No bootstrap required, opened a new log
I20260923 21:01:12.280237 7919 ts_tablet_manager.cc:1397] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20260923 21:01:12.280992 7919 raft_consensus.cc:348] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.281242 7919 raft_consensus.cc:374] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:12.281322 7919 raft_consensus.cc:729] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5abb06795f44babcdb3f3f4040a426, State: Initialized, Role: FOLLOWER
I20260923 21:01:12.281551 7919 consensus_queue.cc:260] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.282425 7919 ts_tablet_manager.cc:1428] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260923 21:01:12.283815 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.282782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:33479)
0923 21:01:12.283398 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.283552 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"threads_started":1}
I20260923 21:01:12.284780 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.284387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:34805)
0923 21:01:12.284581 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.284659 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260923 21:01:12.285322 7668 heartbeater.cc:503] Master 127.0.97.61:33323 was elected leader, sending a full tablet report...
I20260923 21:01:12.481885 7923 raft_consensus.cc:482] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:12.482183 7923 raft_consensus.cc:504] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.483065 7923 leader_election.cc:288] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747), 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:12.483632 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8d930682d4d20aa4fca5ae93377e4" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf0c67eda0742cda4369580b7ba83fa" is_pre_election: true
I20260923 21:01:12.483697 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8d930682d4d20aa4fca5ae93377e4" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5abb06795f44babcdb3f3f4040a426" is_pre_election: true
I20260923 21:01:12.483902 7762 raft_consensus.cc:2454] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 0.
I20260923 21:01:12.483937 7621 raft_consensus.cc:2454] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 0.
I20260923 21:01:12.484448 7408 leader_election.cc:302] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdba6177147a47e693ad2d152127f63a, ddf0c67eda0742cda4369580b7ba83fa; no voters:
I20260923 21:01:12.484779 7923 raft_consensus.cc:2781] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:12.484926 7923 raft_consensus.cc:482] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:12.485019 7923 raft_consensus.cc:3037] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:12.487457 7923 raft_consensus.cc:504] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.488101 7923 leader_election.cc:288] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 election: Requested vote from peers ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747), 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743)
I20260923 21:01:12.488734 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8d930682d4d20aa4fca5ae93377e4" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf0c67eda0742cda4369580b7ba83fa"
I20260923 21:01:12.488991 7762 raft_consensus.cc:3037] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:12.489040 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be8d930682d4d20aa4fca5ae93377e4" candidate_uuid: "cdba6177147a47e693ad2d152127f63a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5abb06795f44babcdb3f3f4040a426"
I20260923 21:01:12.489296 7621 raft_consensus.cc:3037] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:12.491611 7762 raft_consensus.cc:2454] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 1.
I20260923 21:01:12.491580 7621 raft_consensus.cc:2454] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdba6177147a47e693ad2d152127f63a in term 1.
I20260923 21:01:12.492132 7408 leader_election.cc:302] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdba6177147a47e693ad2d152127f63a, ddf0c67eda0742cda4369580b7ba83fa; no voters:
I20260923 21:01:12.492475 7923 raft_consensus.cc:2781] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:12.492975 7923 raft_consensus.cc:686] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 1 LEADER]: Becoming Leader. State: Replica: cdba6177147a47e693ad2d152127f63a, State: Running, Role: LEADER
I20260923 21:01:12.493306 7923 consensus_queue.cc:237] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } }
I20260923 21:01:12.494270 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.493778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:42151)
0923 21:01:12.493963 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.494082 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 21:01:12.496585 7149 catalog_manager.cc:5495] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a reported cstate change: term changed from 0 to 1, leader changed from <none> to cdba6177147a47e693ad2d152127f63a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "cdba6177147a47e693ad2d152127f63a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdba6177147a47e693ad2d152127f63a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 } health_report { overall_health: UNKNOWN } } }
W20260923 21:01:12.519253 7521 tablet.cc:2369] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:12.555224 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.550851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.0.97.1:38547)
0923 21:01:12.551046 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:12.551051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:12.551070 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:01:12.551243 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:12.551247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:12.551315 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:12.551401 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:12.551707 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.552373 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.553650 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.554061 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.554157 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.554597 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:12.554702 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:12.554766 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:12.554873 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 21:01:12.554940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:01:12.558496 7621 raft_consensus.cc:1261] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Refusing update from remote peer cdba6177147a47e693ad2d152127f63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:12.558568 7762 raft_consensus.cc:1261] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Refusing update from remote peer cdba6177147a47e693ad2d152127f63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:12.559217 7927 consensus_queue.cc:1035] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf0c67eda0742cda4369580b7ba83fa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:12.559703 7923 consensus_queue.cc:1035] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5abb06795f44babcdb3f3f4040a426" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:12.562955 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.561846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:51305)
0923 21:01:12.562578 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.562747 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546}
I20260923 21:01:12.563166 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.562587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:41815)
0923 21:01:12.562736 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.562841 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 21:01:12.570300 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.569511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.3:59623)
0923 21:01:12.569936 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.570066 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260923 21:01:12.571388 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.570875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.2:36075)
0923 21:01:12.571135 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.571220 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260923 21:01:12.576182 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.575565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.97.1:37077)
0923 21:01:12.575801 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.575962 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 21:01:12.584141 7148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:12.584463 7148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:12.590557 7148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f90374f37df945d1805e836834117852: Sending DeleteTablet for 3 replicas of tablet 5be8d930682d4d20aa4fca5ae93377e4
I20260923 21:01:12.591717 7453 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be8d930682d4d20aa4fca5ae93377e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:12 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:50150
I20260923 21:01:12.591800 7742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be8d930682d4d20aa4fca5ae93377e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:12 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52906
I20260923 21:01:12.592259 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be8d930682d4d20aa4fca5ae93377e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:12 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:49312
I20260923 21:01:12.592443 7939 tablet_replica.cc:330] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: stopping tablet replica
I20260923 21:01:12.592622 7939 raft_consensus.cc:2229] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:01:12.592958 7941 tablet_replica.cc:330] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: stopping tablet replica
I20260923 21:01:12.593039 7939 raft_consensus.cc:2258] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:12.593262 7941 raft_consensus.cc:2229] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:12.593601 7941 raft_consensus.cc:2258] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:12.594444 7939 ts_tablet_manager.cc:1905] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:12.596249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 7345
I20260923 21:01:12.597574 7941 ts_tablet_manager.cc:1905] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:12.601400 7939 ts_tablet_manager.cc:1918] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:12.601564 7939 log.cc:1196] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/5be8d930682d4d20aa4fca5ae93377e4
I20260923 21:01:12.603127 7939 ts_tablet_manager.cc:1939] T 5be8d930682d4d20aa4fca5ae93377e4 P cdba6177147a47e693ad2d152127f63a: Deleting consensus metadata
I20260923 21:01:12.604082 7940 tablet_replica.cc:330] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: stopping tablet replica
I20260923 21:01:12.604475 7135 catalog_manager.cc:4841] TS cdba6177147a47e693ad2d152127f63a (127.0.97.1:38547): tablet 5be8d930682d4d20aa4fca5ae93377e4 (table test-table [id=aabd2abb1a1343d6845063067866e799]) successfully deleted
I20260923 21:01:12.604939 7941 ts_tablet_manager.cc:1918] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:12.605093 7941 log.cc:1196] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/5be8d930682d4d20aa4fca5ae93377e4
I20260923 21:01:12.605085 7940 raft_consensus.cc:2229] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:12.605510 7941 ts_tablet_manager.cc:1939] T 5be8d930682d4d20aa4fca5ae93377e4 P 5e5abb06795f44babcdb3f3f4040a426: Deleting consensus metadata
I20260923 21:01:12.605490 7940 raft_consensus.cc:2258] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:12.606604 7134 catalog_manager.cc:4841] TS 5e5abb06795f44babcdb3f3f4040a426 (127.0.97.2:41743): tablet 5be8d930682d4d20aa4fca5ae93377e4 (table test-table [id=aabd2abb1a1343d6845063067866e799]) successfully deleted
I20260923 21:01:12.607030 7940 ts_tablet_manager.cc:1905] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:12.613932 7940 ts_tablet_manager.cc:1918] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:12.614106 7940 log.cc:1196] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/5be8d930682d4d20aa4fca5ae93377e4
I20260923 21:01:12.614573 7940 ts_tablet_manager.cc:1939] T 5be8d930682d4d20aa4fca5ae93377e4 P ddf0c67eda0742cda4369580b7ba83fa: Deleting consensus metadata
I20260923 21:01:12.615656 7135 catalog_manager.cc:4841] TS ddf0c67eda0742cda4369580b7ba83fa (127.0.97.3:34747): tablet 5be8d930682d4d20aa4fca5ae93377e4 (table test-table [id=aabd2abb1a1343d6845063067866e799]) successfully deleted
I20260923 21:01:12.625236 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.597667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.0.97.1:38547)
0923 21:01:12.597870 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:12.597879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:12.610052 (+ 12173us) server_negotiation.cc:407] Connection header received
0923 21:01:12.610267 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:12.610273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:12.610395 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:12.610468 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:12.612567 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.613726 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.615277 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.615782 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.615932 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.618880 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:12.618896 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:12.618913 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:12.618950 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:12.622157 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:12.623140 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:12.623150 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:12.623152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:12.623300 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:12.623759 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:12.623763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:12.623765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:12.624155 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:12.624359 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:12.624763 (+ 404us) server_negotiation.cc:299] Negotiation successful
0923 21:01:12.624936 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 21:01:12.626053 7513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:12.676815 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.676265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:39638)
0923 21:01:12.676520 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.676639 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 21:01:12.678232 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.677713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39963 (local address 127.0.0.1:39642)
0923 21:01:12.677933 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:12.678046 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 21:01:12.760111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 7345
I20260923 21:01:12.783344 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 7530
I20260923 21:01:12.797811 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.784133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.0.97.2:41743)
0923 21:01:12.784613 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:12.784625 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:12.784648 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:01:12.784805 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:12.784815 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:12.784881 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:12.785005 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:12.786312 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.787056 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.788320 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.788819 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.789009 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.791917 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:12.791933 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:12.791950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:12.791991 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:12.794884 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:12.795653 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:12.795666 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:12.795669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:12.795802 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:12.796197 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:12.796202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:12.796205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:12.796596 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:12.796905 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:12.797226 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 21:01:12.797453 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":249,"threads_started":1}
I20260923 21:01:12.798744 7661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:12.893883 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 7530
I20260923 21:01:12.911996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 7679
I20260923 21:01:12.927433 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:12.912874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.0.97.3:34747)
0923 21:01:12.913402 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:12.913406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:12.913420 (+ 14us) server_negotiation.cc:407] Connection header received
0923 21:01:12.913478 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:12.913481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:12.913537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:12.913655 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:12.915093 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.915823 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.917138 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:12.917526 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:12.917666 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:12.920392 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:12.920408 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:12.920425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:12.920469 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:12.923799 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:12.924729 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:12.924742 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:12.924744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:12.924961 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:12.925519 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:12.925527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:12.925531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:12.925955 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:12.926303 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:12.926738 (+ 435us) server_negotiation.cc:299] Negotiation successful
0923 21:01:12.927017 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":259,"threads_started":1}
I20260923 21:01:12.928211 7802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:13.028019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 7679
I20260923 21:01:13.045549 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 7012
I20260923 21:01:13.046703 7179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:13.170790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 7012
I20260923 21:01:13.192951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 7109
I20260923 21:01:13.194123 7370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:13.312949 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 7109
2026-09-23T21:01:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12263 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:01:13 dist-test-slave-8xn5 krb5kdc[7952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:01:13 dist-test-slave-8xn5 krb5kdc[7952](info): set up 2 sockets
Sep 23 21:01:13 dist-test-slave-8xn5 krb5kdc[7952](info): commencing operation
krb5kdc: starting...
W20260923 21:01:14.467907 388 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:01:14 dist-test-slave-8xn5 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260923 21:01:14.706772 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33341
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:15.049477 7965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:15.049868 7965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:15.050065 7965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:15.057850 7965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:15.058033 7965 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 21:01:15.058142 7965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:15.058269 7965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:15.058382 7965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:15.058506 7965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:15.069135 7965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33341
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:15.070583 7965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:15.072010 7965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 21:01:15.086475 7965 server_base.cc:1034] running on GCE node
W20260923 21:01:15.086514 7970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:15.086552 7973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:15.086468 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:15.087797 7965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:15.089389 7965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:15.090656 7965 hybrid_clock.cc:648] HybridClock initialized: now 1790197275090588 us; error 54 us; skew 500 ppm
I20260923 21:01:15.091044 7965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:15 dist-test-slave-8xn5 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:15.102630 7965 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260923 21:01:15.104364 7965 webserver.cc:460] Webserver started at http://127.0.97.62:44193/ using document root <none> and password file <none>
I20260923 21:01:15.105175 7965 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:15.105319 7965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:15.105667 7965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:15.108950 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "0e398c89678148f9b636c9bc43dadb29"
format_stamp: "Formatted at 2026-09-23 21:01:15 on dist-test-slave-8xn5"
server_key: "d59bc5398841373e2ba6d1f240ce8b19"
server_key_iv: "7198662e6cba1cc6f0b5b9b31a995e95"
server_key_version: "encryptionkey@0"
I20260923 21:01:15.109849 7965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "0e398c89678148f9b636c9bc43dadb29"
format_stamp: "Formatted at 2026-09-23 21:01:15 on dist-test-slave-8xn5"
server_key: "d59bc5398841373e2ba6d1f240ce8b19"
server_key_iv: "7198662e6cba1cc6f0b5b9b31a995e95"
server_key_version: "encryptionkey@0"
I20260923 21:01:15.115581 7965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 21:01:15.119449 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:15.121078 7965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:01:15.121250 7965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "0e398c89678148f9b636c9bc43dadb29"
format_stamp: "Formatted at 2026-09-23 21:01:15 on dist-test-slave-8xn5"
server_key: "d59bc5398841373e2ba6d1f240ce8b19"
server_key_iv: "7198662e6cba1cc6f0b5b9b31a995e95"
server_key_version: "encryptionkey@0"
I20260923 21:01:15.121454 7965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:15.135690 7965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:15.140280 7965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:15.140534 7965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:15.164628 7965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33341
I20260923 21:01:15.164647 8041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33341 every 8 connection(s)
I20260923 21:01:15.166106 7965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:01:15.170482 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:15.170483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 7965
I20260923 21:01:15.170681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:01:15.171478 388 external_mini_cluster.cc:1428] Setting key ffb1ef13a26b1d14018cfbd86ae4a133
I20260923 21:01:15.185446 8042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Bootstrap starting.
Sep 23 21:01:15 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:15.189404 8042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:15.190598 8042 log.cc:824] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:15.193641 8042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: No bootstrap required, opened a new log
I20260923 21:01:15.197974 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:15.173761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.0.97.62:33341)
0923 21:01:15.174632 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:15.174645 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:15.174688 (+ 43us) server_negotiation.cc:407] Connection header received
0923 21:01:15.177035 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:15.177063 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:15.177526 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:15.177913 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:15.179760 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.181239 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.183184 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.184019 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.184165 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:15.189413 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:15.189422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:15.189440 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:15.189491 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:15.193151 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:15.194074 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:15.194082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:15.194090 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:15.194289 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:15.194807 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:15.194811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:15.194814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:15.195405 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:15.195773 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:15.196296 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 21:01:15.196802 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":403,"threads_started":1}
I20260923 21:01:15.202714 8042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e398c89678148f9b636c9bc43dadb29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260923 21:01:15.203143 8042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:15.203269 8042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e398c89678148f9b636c9bc43dadb29, State: Initialized, Role: FOLLOWER
I20260923 21:01:15.203909 8042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e398c89678148f9b636c9bc43dadb29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260923 21:01:15.204157 8042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:01:15.204304 8042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:01:15.204488 8042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:15.207165 8042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e398c89678148f9b636c9bc43dadb29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260923 21:01:15.207765 8042 leader_election.cc:302] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e398c89678148f9b636c9bc43dadb29; no voters:
I20260923 21:01:15.208350 8042 leader_election.cc:288] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:01:15.208527 8047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:15.209425 8047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [term 1 LEADER]: Becoming Leader. State: Replica: 0e398c89678148f9b636c9bc43dadb29, State: Running, Role: LEADER
I20260923 21:01:15.210135 8047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e398c89678148f9b636c9bc43dadb29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } }
I20260923 21:01:15.212641 8042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:15.214424 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e398c89678148f9b636c9bc43dadb29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e398c89678148f9b636c9bc43dadb29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } } }
I20260923 21:01:15.214816 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:15.216361 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e398c89678148f9b636c9bc43dadb29. Latest consensus state: current_term: 1 leader_uuid: "0e398c89678148f9b636c9bc43dadb29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e398c89678148f9b636c9bc43dadb29" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33341 } } }
I20260923 21:01:15.217051 8052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:15.218709 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:15.229008 8052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:15.248319 8052 catalog_manager.cc:1347] Generated new cluster ID: 195ad3d9bbea4a5b86cc1bf3d0fc35c5
I20260923 21:01:15.248473 8052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:15.266903 8052 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:01:15.268290 8052 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:15.284852 8052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Generated new TSK 0
I20260923 21:01:15.285820 8052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260923 21:01:15.355044 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:15.710379 8068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:15.710724 8068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:15.710887 8068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:15.718719 8068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:15.718879 8068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:15.719029 8068 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:01:15.719163 8068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:15.729902 8068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:15.731383 8068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:15.732860 8068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:15.747366 8076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:15.747465 8074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:15.747349 8073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:15.752427 8068 server_base.cc:1034] running on GCE node
I20260923 21:01:15.753182 8068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:15.754287 8068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:15.755602 8068 hybrid_clock.cc:648] HybridClock initialized: now 1790197275755505 us; error 75 us; skew 500 ppm
I20260923 21:01:15.755942 8068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:15 dist-test-slave-8xn5 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:15.759397 8068 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 21:01:15.761210 8068 webserver.cc:460] Webserver started at http://127.0.97.1:36211/ using document root <none> and password file <none>
I20260923 21:01:15.762007 8068 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:15.762152 8068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:15.762573 8068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:15.765763 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "a997ec2c9ee4471fba5dc195544e2cba"
format_stamp: "Formatted at 2026-09-23 21:01:15 on dist-test-slave-8xn5"
server_key: "8f2830734048777e3727de1f5f7feb10"
server_key_iv: "dd099228391f80f5d7c9274bc01cafc8"
server_key_version: "encryptionkey@0"
I20260923 21:01:15.766779 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a997ec2c9ee4471fba5dc195544e2cba"
format_stamp: "Formatted at 2026-09-23 21:01:15 on dist-test-slave-8xn5"
server_key: "8f2830734048777e3727de1f5f7feb10"
server_key_iv: "dd099228391f80f5d7c9274bc01cafc8"
server_key_version: "encryptionkey@0"
I20260923 21:01:15.772708 8068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:01:15.777187 8083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:15.779202 8068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:01:15.779415 8068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "a997ec2c9ee4471fba5dc195544e2cba"
format_stamp: "Formatted at 2026-09-23 21:01:15 on dist-test-slave-8xn5"
server_key: "8f2830734048777e3727de1f5f7feb10"
server_key_iv: "dd099228391f80f5d7c9274bc01cafc8"
server_key_version: "encryptionkey@0"
I20260923 21:01:15.779686 8068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:15.807638 8068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:15.812588 8068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:15.812945 8068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:15.816038 8068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:15.816159 8068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:15.816331 8068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:15.816433 8068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:01:15 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:15.845254 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:15.816564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60533 (local address 127.0.97.62:33341)
0923 21:01:15.817369 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:15.817376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:15.818592 (+ 1216us) server_negotiation.cc:407] Connection header received
0923 21:01:15.820713 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:15.820719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:15.820846 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:15.820983 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:15.823989 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.825394 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.827800 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.828441 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.828608 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:15.835119 (+ 6511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:15.835125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:15.835128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:15.835165 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:15.838631 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:15.840084 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:15.840093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:15.840100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:15.840275 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:15.840872 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:15.840878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:15.840881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:15.841357 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:15.841550 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:15.844687 (+ 3137us) server_negotiation.cc:299] Negotiation successful
0923 21:01:15.844926 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":546,"threads_started":1}
I20260923 21:01:15.845498 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:15.817369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.1:60533)
0923 21:01:15.818379 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:15.818441 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:15.820388 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:15.821529 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:15.821538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:15.822549 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:15.823626 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:15.823678 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.825655 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.825666 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:15.827611 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:15.827624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.828746 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.828756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:15.829164 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:15.830384 (+ 1220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:15.830431 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:15.834802 (+ 4371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:15.838826 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:15.838837 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:15.838858 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:15.839897 (+ 1039us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:15.840422 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:15.840427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:15.840435 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:15.840673 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:15.841633 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:15.841645 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:15.842304 (+ 659us) client_negotiation.cc:769] Sending connection context
0923 21:01:15.842764 (+ 460us) client_negotiation.cc:240] Negotiation successful
0923 21:01:15.843579 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":401,"threads_started":1}
I20260923 21:01:15.860092 8068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34429
I20260923 21:01:15.860227 8203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34429 every 8 connection(s)
I20260923 21:01:15.861567 8068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:15.862063 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 8068
I20260923 21:01:15.862272 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:01:15.863188 388 external_mini_cluster.cc:1428] Setting key a5021a596a625d541d0df4357555c13a
I20260923 21:01:15.865854 8204 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33341
I20260923 21:01:15.866214 8204 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:15.867360 8204 heartbeater.cc:511] Master 127.0.97.62:33341 requested a full tablet report, sending...
I20260923 21:01:15.869977 7997 ts_manager.cc:194] Registered new tserver with Master: a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429)
I20260923 21:01:15.871770 7997 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60533
I20260923 21:01:15.875121 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:15.855163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44589 (local address 127.0.97.62:33341)
0923 21:01:15.855423 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:15.855435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:15.856954 (+ 1519us) server_negotiation.cc:407] Connection header received
0923 21:01:15.857122 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:15.857131 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:15.857243 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:15.857383 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:15.859048 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.860263 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.862449 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.863259 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.863409 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:15.866598 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:15.866607 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:15.866613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:15.866655 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:15.871113 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:15.872181 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:15.872186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:15.872190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:15.872355 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:15.872898 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:15.872906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:15.872913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:15.873357 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:15.873534 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:15.874554 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0923 21:01:15.874791 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 21:01:15.875211 8196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:15.855839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.1:44589)
0923 21:01:15.856821 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:15.856844 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:15.857004 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:15.857617 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:15.857625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:15.858359 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:15.858789 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:15.858809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.860473 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.860478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:15.861985 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:15.862005 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:15.863834 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:15.863844 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:15.864051 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:15.864872 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:15.864897 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:15.866225 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:15.871365 (+ 5140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:15.871374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:15.871397 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:15.872032 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:15.872516 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:15.872525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:15.872532 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:15.872727 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:15.873615 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:15.873624 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:15.873874 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 21:01:15.874050 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 21:01:15.874584 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":574,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260923 21:01:15.917924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:16.264178 8208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:16.264496 8208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:16.264596 8208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:16.272107 8208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:16.272239 8208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:16.272324 8208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:01:16.272406 8208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:16.284700 8208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:16.286128 8208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:16.287631 8208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:16.301468 8213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:16.301565 8214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:16.302004 8216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:16.302568 8208 server_base.cc:1034] running on GCE node
I20260923 21:01:16.303232 8208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:16.304390 8208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:16.305697 8208 hybrid_clock.cc:648] HybridClock initialized: now 1790197276305610 us; error 57 us; skew 500 ppm
I20260923 21:01:16.306134 8208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:16 dist-test-slave-8xn5 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:16.310120 8208 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 21:01:16.311980 8208 webserver.cc:460] Webserver started at http://127.0.97.2:32905/ using document root <none> and password file <none>
I20260923 21:01:16.312774 8208 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:16.312945 8208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:16.313357 8208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:16.317072 8208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "22220f420436443caa4bab5f96c8a5ec"
format_stamp: "Formatted at 2026-09-23 21:01:16 on dist-test-slave-8xn5"
server_key: "492fc66e63a652e9b9ac6e2f74a2f05a"
server_key_iv: "3afa376287eb4e99425b06327baf967b"
server_key_version: "encryptionkey@0"
I20260923 21:01:16.318042 8208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "22220f420436443caa4bab5f96c8a5ec"
format_stamp: "Formatted at 2026-09-23 21:01:16 on dist-test-slave-8xn5"
server_key: "492fc66e63a652e9b9ac6e2f74a2f05a"
server_key_iv: "3afa376287eb4e99425b06327baf967b"
server_key_version: "encryptionkey@0"
I20260923 21:01:16.325136 8208 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260923 21:01:16.329120 8223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:16.330879 8208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:01:16.331081 8208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "22220f420436443caa4bab5f96c8a5ec"
format_stamp: "Formatted at 2026-09-23 21:01:16 on dist-test-slave-8xn5"
server_key: "492fc66e63a652e9b9ac6e2f74a2f05a"
server_key_iv: "3afa376287eb4e99425b06327baf967b"
server_key_version: "encryptionkey@0"
I20260923 21:01:16.331403 8208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:16.346685 8208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:16.351343 8208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:16.351805 8208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:16.355115 8208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:16.355218 8208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:16.355427 8208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:16.355480 8208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:01:16 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:16.385833 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.356042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36517 (local address 127.0.97.62:33341)
0923 21:01:16.356292 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:16.356304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:16.358019 (+ 1715us) server_negotiation.cc:407] Connection header received
0923 21:01:16.362111 (+ 4092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:16.362118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:16.362210 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:16.362306 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:16.365397 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.366134 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.370442 (+ 4308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.370903 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.371079 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.377380 (+ 6301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:16.377392 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:16.377397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:16.377464 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:16.380755 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.382428 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.382435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.382438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.382581 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.383501 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.383510 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.383515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.383939 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:16.384112 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:16.385221 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0923 21:01:16.385479 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 21:01:16.387243 8232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.356712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.2:36517)
0923 21:01:16.357793 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:16.357849 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:16.361652 (+ 3803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:16.362732 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:16.362751 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:16.363882 (+ 1131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:16.365070 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.365108 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.366362 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.366372 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.370071 (+ 3699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.370104 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.371288 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.371302 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.371680 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.372572 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:16.372602 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:16.376985 (+ 4383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:16.380989 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.381007 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.381033 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.382206 (+ 1173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.382719 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.382726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.382732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.383157 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.384117 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:16.384132 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:16.384741 (+ 609us) client_negotiation.cc:769] Sending connection context
0923 21:01:16.385104 (+ 363us) client_negotiation.cc:240] Negotiation successful
0923 21:01:16.385682 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":373,"threads_started":1}
I20260923 21:01:16.408927 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.393605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35065 (local address 127.0.97.62:33341)
0923 21:01:16.393811 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:16.393819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:16.394750 (+ 931us) server_negotiation.cc:407] Connection header received
0923 21:01:16.395008 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:16.395013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:16.395103 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:16.395220 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:16.396141 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.396886 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.398228 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.398717 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.398886 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.401536 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:16.401544 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:16.401549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:16.401612 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:16.405922 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.406883 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.406892 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.406896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.407028 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.407418 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.407425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.407429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.407772 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:16.407941 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:16.408351 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 21:01:16.408568 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:01:16.409066 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.393626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.2:35065)
0923 21:01:16.394605 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:16.394640 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:16.394851 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:16.395244 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:16.395249 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:16.395712 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:16.395997 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.396007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.397034 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.397039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.398060 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.398071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.399149 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.399155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.399298 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.400126 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:16.400143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:16.401279 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:16.406161 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.406170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.406180 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.406738 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.407149 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.407154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.407161 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.407297 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.407905 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:16.407910 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:16.408099 (+ 189us) client_negotiation.cc:769] Sending connection context
0923 21:01:16.408244 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 21:01:16.408598 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":493,"threads_started":1}
I20260923 21:01:16.421169 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32775 every 8 connection(s)
I20260923 21:01:16.421208 8208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32775
I20260923 21:01:16.422763 8208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:16.425063 8343 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33341
I20260923 21:01:16.425336 8343 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:16.426110 8343 heartbeater.cc:511] Master 127.0.97.62:33341 requested a full tablet report, sending...
I20260923 21:01:16.427577 7997 ts_manager.cc:194] Registered new tserver with Master: 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:16.428412 7997 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36517
I20260923 21:01:16.431557 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 8208
I20260923 21:01:16.431686 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:01:16.432102 388 external_mini_cluster.cc:1428] Setting key 6305ec44498c78c3938644055e88da70
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 21:01:16.486258 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33341
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:16.849059 8347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:16.849371 8347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:16.849471 8347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:16.856997 8347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:16.857136 8347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:16.857224 8347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 21:01:16.857306 8347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:16.867982 8347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:16.869262 8347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:16.870851 8347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 21:01:16.875432 8204 heartbeater.cc:503] Master 127.0.97.62:33341 was elected leader, sending a full tablet report...
W20260923 21:01:16.885779 8353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:16.885732 8352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:16.885902 8355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:16.886698 8347 server_base.cc:1034] running on GCE node
I20260923 21:01:16.887439 8347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:16.888653 8347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:16.889931 8347 hybrid_clock.cc:648] HybridClock initialized: now 1790197276889828 us; error 67 us; skew 500 ppm
I20260923 21:01:16.890288 8347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:16 dist-test-slave-8xn5 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:16.893690 8347 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 21:01:16.895292 8347 webserver.cc:460] Webserver started at http://127.0.97.3:39991/ using document root <none> and password file <none>
I20260923 21:01:16.896138 8347 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:16.896293 8347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:16.896656 8347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:16.899961 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "d37c7cd1ec92414986ab789cea413034"
format_stamp: "Formatted at 2026-09-23 21:01:16 on dist-test-slave-8xn5"
server_key: "f3a728b203065bc2066acce851b19af6"
server_key_iv: "81d5a5db607b7582c66ae93f6dde40b2"
server_key_version: "encryptionkey@0"
I20260923 21:01:16.900843 8347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d37c7cd1ec92414986ab789cea413034"
format_stamp: "Formatted at 2026-09-23 21:01:16 on dist-test-slave-8xn5"
server_key: "f3a728b203065bc2066acce851b19af6"
server_key_iv: "81d5a5db607b7582c66ae93f6dde40b2"
server_key_version: "encryptionkey@0"
I20260923 21:01:16.906996 8347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260923 21:01:16.911271 8362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:16.912946 8347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 21:01:16.913122 8347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "d37c7cd1ec92414986ab789cea413034"
format_stamp: "Formatted at 2026-09-23 21:01:16 on dist-test-slave-8xn5"
server_key: "f3a728b203065bc2066acce851b19af6"
server_key_iv: "81d5a5db607b7582c66ae93f6dde40b2"
server_key_version: "encryptionkey@0"
I20260923 21:01:16.913406 8347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:16.928176 8347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:16.932755 8347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:16.933079 8347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:16.936080 8347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:16.936218 8347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:16.936384 8347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:16.936453 8347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:01:16 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197276, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:16.963433 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.937086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33473 (local address 127.0.97.62:33341)
0923 21:01:16.937550 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:16.937557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:16.938937 (+ 1380us) server_negotiation.cc:407] Connection header received
0923 21:01:16.941803 (+ 2866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:16.941808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:16.941907 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:16.942042 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:16.944461 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.945792 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.948692 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.949399 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.949559 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.954926 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:16.954933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:16.954937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:16.954975 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:16.958586 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.959818 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.959827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.959832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.960054 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.960937 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.960942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.960946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.961407 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:16.961569 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:16.962763 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0923 21:01:16.963013 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260923 21:01:16.964627 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.937794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.3:33473)
0923 21:01:16.938705 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:16.938764 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:16.941442 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:16.942385 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:16.942402 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:16.943319 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:16.944217 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.944239 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.946031 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.946039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.948455 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.948513 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.949674 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.949681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.950015 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.950996 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:16.951040 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:16.954603 (+ 3563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:16.958760 (+ 4157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.958770 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.958807 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.959598 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.960206 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.960211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.960215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.960772 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.961543 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:16.961551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:16.962202 (+ 651us) client_negotiation.cc:769] Sending connection context
0923 21:01:16.962665 (+ 463us) client_negotiation.cc:240] Negotiation successful
0923 21:01:16.963317 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":392,"threads_started":1}
I20260923 21:01:16.985756 8347 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36127
I20260923 21:01:16.985757 8482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36127 every 8 connection(s)
I20260923 21:01:16.988152 8347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:16.989642 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.972320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48693 (local address 127.0.97.62:33341)
0923 21:01:16.972562 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:16.972571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:16.973276 (+ 705us) server_negotiation.cc:407] Connection header received
0923 21:01:16.973517 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:16.973523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:16.973630 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:16.973802 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:16.975496 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.976630 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.979019 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.979516 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.979694 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.981876 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:16.981882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:16.981885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:16.981919 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:16.985693 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.987337 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.987343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.987346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.987493 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:16.987972 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:16.987976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:16.987978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:16.988372 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:16.988536 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:16.988981 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 21:01:16.989229 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 21:01:16.989728 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:16.972323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.97.3:48693)
0923 21:01:16.973122 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:16.973149 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:16.973357 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:16.973834 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:16.973840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:16.974666 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:16.975279 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.975297 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.976836 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.976840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.978830 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:16.978846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:16.979737 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:16.979745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:16.979923 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:16.980767 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:16.980787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:16.981627 (+ 840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:16.985869 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.985879 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.985883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.986415 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.987627 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:16.987636 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:16.987640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:16.987828 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:16.988496 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:16.988500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:16.988711 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 21:01:16.988868 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 21:01:16.989238 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":432,"threads_started":1}
I20260923 21:01:16.990602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 8347
I20260923 21:01:16.990813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:01:16.991487 388 external_mini_cluster.cc:1428] Setting key d98d0298292c71e82c40e6c27b9bb0dc
I20260923 21:01:16.992692 8483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33341
I20260923 21:01:16.993053 8483 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:16.994310 8483 heartbeater.cc:511] Master 127.0.97.62:33341 requested a full tablet report, sending...
I20260923 21:01:16.996214 7996 ts_manager.cc:194] Registered new tserver with Master: d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127)
I20260923 21:01:16.997151 7996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33473
I20260923 21:01:17.008924 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 21:01:17 dist-test-slave-8xn5 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197277, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 21:01:17 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:17 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 21:01:17.063701 8371 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:17.052743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.0.97.62:33341)
0923 21:01:17.052941 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.052946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.052970 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:01:17.053197 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.053202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.053270 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.053465 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:17.054943 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.055741 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.057027 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.057433 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.057533 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.063461 (+ 5928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48078: BlockingRecv error: recv got EOF from 127.0.0.1:48078 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 21:01:17 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197277, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:17.083649 8371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.067085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.97.62:33341)
0923 21:01:17.067311 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.067317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.067887 (+ 570us) server_negotiation.cc:407] Connection header received
0923 21:01:17.068034 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.068048 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.068114 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.068209 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:17.069593 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.070320 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.072028 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.072494 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.072602 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.077315 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:17.077321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:17.077325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:17.077369 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:17.080438 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:17.081229 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:17.081234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:17.081236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:17.081492 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:17.081934 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:17.081937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:17.081939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:17.082216 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:17.082615 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.082906 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.083142 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:01:17.088815 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:01:17.096938 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:17.136548 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.114774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.0.97.2:32775)
0923 21:01:17.125125 (+ 10351us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.125138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.127219 (+ 2081us) server_negotiation.cc:407] Connection header received
0923 21:01:17.127500 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.127510 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.127787 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.128045 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:17.129364 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.130787 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.133996 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.135157 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.135335 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.135571 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.136020 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.136211 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10113,"thread_start_us":9920,"threads_started":1}
I20260923 21:01:17.136730 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.114504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32775 (local address 127.0.0.1:55512)
0923 21:01:17.126869 (+ 12365us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.126892 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.127039 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.128045 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.128055 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.128089 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:17.129130 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.129144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.130930 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.130937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.133774 (+ 2837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.133790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.135407 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.135414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.135622 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.135779 (+ 157us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.135949 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.136273 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12218,"mutex_wait_us":201,"thread_start_us":7156,"threads_started":1}
I20260923 21:01:17.136981 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.113298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.97.3:36127)
0923 21:01:17.126449 (+ 13151us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.126463 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.126579 (+ 116us) server_negotiation.cc:407] Connection header received
0923 21:01:17.126710 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.126720 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.127117 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.127341 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:17.129193 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.130484 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.133970 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.135462 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.135577 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.135779 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.136476 (+ 697us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.136658 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12890,"thread_start_us":12656,"threads_started":1}
I20260923 21:01:17.137782 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.112589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.0.1:33956)
0923 21:01:17.125848 (+ 13259us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.125882 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.126079 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.127655 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.127662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.127715 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:17.128990 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.129000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.130646 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.130652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.133775 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.133790 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.135980 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.135984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.136164 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.136276 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.136366 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.137375 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13033,"mutex_wait_us":1069,"spinlock_wait_cycles":896,"thread_start_us":12844,"threads_started":1}
I20260923 21:01:17.138518 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.112228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34429 (local address 127.0.0.1:51628)
0923 21:01:17.123036 (+ 10808us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.123100 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.123428 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.127553 (+ 4125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.127561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.127596 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:17.128375 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.128390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.130609 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.130617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.133463 (+ 2846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.133480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.135969 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.135976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.136165 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.136276 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.136366 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.136524 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9689,"mutex_wait_us":248,"spinlock_wait_cycles":896,"thread_start_us":14291,"threads_started":1}
I20260923 21:01:17.139077 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.112674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.0.97.1:34429)
0923 21:01:17.124539 (+ 11865us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.124553 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.124599 (+ 46us) server_negotiation.cc:407] Connection header received
0923 21:01:17.124743 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.124751 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.126393 (+ 1642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.127228 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:17.128763 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.130450 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.133671 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.135346 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.135499 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.135750 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.138534 (+ 2784us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.138793 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11658,"thread_start_us":11502,"threads_started":1}
I20260923 21:01:17.141546 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9fd19f86f84a8eafd38944fdf3e285 (DEFAULT_TABLE table=test-table [id=fc9d38c33eca4b4bbaefcbdec9ef6898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:17.141506 8273 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9fd19f86f84a8eafd38944fdf3e285 (DEFAULT_TABLE table=test-table [id=fc9d38c33eca4b4bbaefcbdec9ef6898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:17.141983 8134 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9fd19f86f84a8eafd38944fdf3e285 (DEFAULT_TABLE table=test-table [id=fc9d38c33eca4b4bbaefcbdec9ef6898]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:17.143168 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9fd19f86f84a8eafd38944fdf3e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.143172 8273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9fd19f86f84a8eafd38944fdf3e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.143290 8134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9fd19f86f84a8eafd38944fdf3e285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.156189 8507 tablet_bootstrap.cc:492] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Bootstrap starting.
I20260923 21:01:17.159870 8507 tablet_bootstrap.cc:654] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.161365 8507 log.cc:824] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:17.164304 8508 tablet_bootstrap.cc:492] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Bootstrap starting.
I20260923 21:01:17.165047 8507 tablet_bootstrap.cc:492] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: No bootstrap required, opened a new log
I20260923 21:01:17.165364 8507 ts_tablet_manager.cc:1397] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 21:01:17.167840 8508 tablet_bootstrap.cc:654] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.169039 8508 log.cc:824] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:17.172430 8508 tablet_bootstrap.cc:492] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: No bootstrap required, opened a new log
I20260923 21:01:17.172868 8508 ts_tablet_manager.cc:1397] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260923 21:01:17.174719 8509 tablet_bootstrap.cc:492] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Bootstrap starting.
I20260923 21:01:17.177282 8507 raft_consensus.cc:348] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.178113 8507 raft_consensus.cc:374] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.178285 8507 raft_consensus.cc:729] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a997ec2c9ee4471fba5dc195544e2cba, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.179154 8509 tablet_bootstrap.cc:654] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.179152 8507 consensus_queue.cc:260] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.180791 8509 log.cc:824] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:17.185407 8508 raft_consensus.cc:348] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.186146 8508 raft_consensus.cc:374] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.186375 8508 raft_consensus.cc:729] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.187599 8508 consensus_queue.cc:260] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.190378 8507 ts_tablet_manager.cc:1428] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Time spent starting tablet: real 0.025s user 0.021s sys 0.005s
I20260923 21:01:17.191954 8483 heartbeater.cc:503] Master 127.0.97.62:33341 was elected leader, sending a full tablet report...
I20260923 21:01:17.193333 8508 ts_tablet_manager.cc:1428] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Time spent starting tablet: real 0.020s user 0.007s sys 0.007s
I20260923 21:01:17.196478 8509 tablet_bootstrap.cc:492] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: No bootstrap required, opened a new log
I20260923 21:01:17.196859 8509 ts_tablet_manager.cc:1397] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.008s
I20260923 21:01:17.204967 8509 raft_consensus.cc:348] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.205554 8509 raft_consensus.cc:374] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.205720 8509 raft_consensus.cc:729] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22220f420436443caa4bab5f96c8a5ec, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.206509 8509 consensus_queue.cc:260] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.207669 8343 heartbeater.cc:503] Master 127.0.97.62:33341 was elected leader, sending a full tablet report...
I20260923 21:01:17.208405 8509 ts_tablet_manager.cc:1428] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
W20260923 21:01:17.246177 8484 tablet.cc:2369] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:17.312196 8512 raft_consensus.cc:482] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:17.312670 8512 raft_consensus.cc:504] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.314602 8512 leader_election.cc:288] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:17.323036 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.315164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49581 (local address 127.0.97.3:36127)
0923 21:01:17.315375 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.315381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.316477 (+ 1096us) server_negotiation.cc:407] Connection header received
0923 21:01:17.316690 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.316698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.316795 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.316863 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:17.317799 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.318861 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.320811 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.321843 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.322004 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.322141 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.322638 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.322765 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:01:17.323045 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.315223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55535 (local address 127.0.97.2:32775)
0923 21:01:17.315426 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.315433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.316084 (+ 651us) server_negotiation.cc:407] Connection header received
0923 21:01:17.316344 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.316349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.316411 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.316531 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:17.317168 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.318225 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.320408 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.321817 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.322013 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.322132 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.322604 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.322750 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:01:17.323220 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.315058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32775 (local address 127.0.97.1:55535)
0923 21:01:17.315956 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.315973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.316176 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.316573 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.316580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.316607 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:17.316999 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.317013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.318392 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.318398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.320176 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.320190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.322145 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.322153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.322404 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.322459 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.322544 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.322720 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":67,"thread_start_us":551,"threads_started":1}
I20260923 21:01:17.323251 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.315646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.97.1:49581)
0923 21:01:17.316342 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.316361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.316506 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.317148 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.317152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.317177 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:17.317559 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.317570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.319009 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.319013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.320653 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.320669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.322108 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.322117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.322451 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.322476 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.322592 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.322747 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":37,"thread_start_us":395,"threads_started":1}
I20260923 21:01:17.324415 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9fd19f86f84a8eafd38944fdf3e285" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec" is_pre_election: true
I20260923 21:01:17.324334 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9fd19f86f84a8eafd38944fdf3e285" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37c7cd1ec92414986ab789cea413034" is_pre_election: true
I20260923 21:01:17.324977 8433 raft_consensus.cc:2454] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 0.
I20260923 21:01:17.324991 8298 raft_consensus.cc:2454] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 0.
I20260923 21:01:17.325882 8085 leader_election.cc:302] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, a997ec2c9ee4471fba5dc195544e2cba; no voters:
I20260923 21:01:17.326666 8512 raft_consensus.cc:2781] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:17.326851 8512 raft_consensus.cc:482] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:17.326961 8512 raft_consensus.cc:3037] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.329274 8512 raft_consensus.cc:504] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.331533 8512 leader_election.cc:288] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 election: Requested vote from peers d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:17.332345 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9fd19f86f84a8eafd38944fdf3e285" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37c7cd1ec92414986ab789cea413034"
I20260923 21:01:17.332340 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9fd19f86f84a8eafd38944fdf3e285" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec"
I20260923 21:01:17.332614 8298 raft_consensus.cc:3037] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.332613 8433 raft_consensus.cc:3037] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.336248 8298 raft_consensus.cc:2454] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 1.
I20260923 21:01:17.336248 8433 raft_consensus.cc:2454] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 1.
I20260923 21:01:17.336817 8085 leader_election.cc:302] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, a997ec2c9ee4471fba5dc195544e2cba, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:17.337251 8512 raft_consensus.cc:2781] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:17.337872 8512 raft_consensus.cc:686] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 1 LEADER]: Becoming Leader. State: Replica: a997ec2c9ee4471fba5dc195544e2cba, State: Running, Role: LEADER
I20260923 21:01:17.338493 8512 consensus_queue.cc:237] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } }
I20260923 21:01:17.343504 7996 catalog_manager.cc:5495] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba reported cstate change: term changed from 0 to 1, leader changed from <none> to a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a997ec2c9ee4471fba5dc195544e2cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } health_report { overall_health: UNKNOWN } } }
W20260923 21:01:17.370236 8205 tablet.cc:2369] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:17.377028 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.371509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.0.97.1:34429)
0923 21:01:17.371697 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.371701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.371757 (+ 56us) server_negotiation.cc:407] Connection header received
0923 21:01:17.371964 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.371968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.372056 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.372191 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:17.372504 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.373200 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.374656 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.375054 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.375154 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.375855 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:17.376500 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:17.376619 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.376736 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.376812 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:01:17.386904 8433 raft_consensus.cc:1261] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Refusing update from remote peer a997ec2c9ee4471fba5dc195544e2cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:17.386904 8298 raft_consensus.cc:1261] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Refusing update from remote peer a997ec2c9ee4471fba5dc195544e2cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:17.388094 8512 consensus_queue.cc:1035] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:17.388482 8518 consensus_queue.cc:1035] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:17.399226 8522 mvcc.cc:205] Tried to move back new op lower bound from 7332648048164433920 to 7332648047982350336. Current Snapshot: MvccSnapshot[applied={T|T < 7332648048164433920}]
I20260923 21:01:17.421562 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:17.421893 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:17.426501 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Sending DeleteTablet for 3 replicas of tablet 8a9fd19f86f84a8eafd38944fdf3e285
I20260923 21:01:17.427914 8134 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9fd19f86f84a8eafd38944fdf3e285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51628
I20260923 21:01:17.428149 8273 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9fd19f86f84a8eafd38944fdf3e285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 21:01:17.429469 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9fd19f86f84a8eafd38944fdf3e285 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:17 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260923 21:01:17.440413 7996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:01:17.443773 7996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:17.446924 8532 tablet_replica.cc:330] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: stopping tablet replica
I20260923 21:01:17.447469 8532 raft_consensus.cc:2229] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:17.448069 8532 raft_consensus.cc:2258] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:17.450834 8533 tablet_replica.cc:330] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: stopping tablet replica
I20260923 21:01:17.450954 8532 ts_tablet_manager.cc:1905] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:17.451303 8533 raft_consensus.cc:2229] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:17.451809 8533 raft_consensus.cc:2258] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:17.452628 8531 tablet_replica.cc:330] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: stopping tablet replica
I20260923 21:01:17.453078 8531 raft_consensus.cc:2229] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:01:17.453769 8531 raft_consensus.cc:2258] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:17.456614 8533 ts_tablet_manager.cc:1905] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:17.458261 8133 tablet_service.cc:1467] Processing CreateTablet for tablet c02c58cf3597414196c4f1b12ae3c4fa (DEFAULT_TABLE table=test-table [id=aabd5a326e7e429ab08079f51abb7bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:17.458819 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02c58cf3597414196c4f1b12ae3c4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.458791 8413 tablet_service.cc:1467] Processing CreateTablet for tablet c02c58cf3597414196c4f1b12ae3c4fa (DEFAULT_TABLE table=test-table [id=aabd5a326e7e429ab08079f51abb7bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:17.459251 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02c58cf3597414196c4f1b12ae3c4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.465624 8531 ts_tablet_manager.cc:1905] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:17.467046 8507 tablet_bootstrap.cc:492] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: Bootstrap starting.
I20260923 21:01:17.469098 8507 tablet_bootstrap.cc:654] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.469969 8508 tablet_bootstrap.cc:492] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: Bootstrap starting.
I20260923 21:01:17.470033 8533 ts_tablet_manager.cc:1918] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:17.470224 8533 log.cc:1196] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/8a9fd19f86f84a8eafd38944fdf3e285
I20260923 21:01:17.470824 8533 ts_tablet_manager.cc:1939] T 8a9fd19f86f84a8eafd38944fdf3e285 P d37c7cd1ec92414986ab789cea413034: Deleting consensus metadata
I20260923 21:01:17.471109 8532 ts_tablet_manager.cc:1918] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:17.471288 8532 log.cc:1196] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/8a9fd19f86f84a8eafd38944fdf3e285
I20260923 21:01:17.471696 8508 tablet_bootstrap.cc:654] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.471858 8532 ts_tablet_manager.cc:1939] T 8a9fd19f86f84a8eafd38944fdf3e285 P 22220f420436443caa4bab5f96c8a5ec: Deleting consensus metadata
I20260923 21:01:17.473100 7981 catalog_manager.cc:4841] TS d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127): tablet 8a9fd19f86f84a8eafd38944fdf3e285 (table test-table [id=fc9d38c33eca4b4bbaefcbdec9ef6898]) successfully deleted
I20260923 21:01:17.473196 8273 tablet_service.cc:1467] Processing CreateTablet for tablet c02c58cf3597414196c4f1b12ae3c4fa (DEFAULT_TABLE table=test-table [id=aabd5a326e7e429ab08079f51abb7bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:17.473667 8273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02c58cf3597414196c4f1b12ae3c4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.473933 7982 catalog_manager.cc:4841] TS 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775): tablet 8a9fd19f86f84a8eafd38944fdf3e285 (table test-table [id=fc9d38c33eca4b4bbaefcbdec9ef6898]) successfully deleted
I20260923 21:01:17.475994 8508 tablet_bootstrap.cc:492] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: No bootstrap required, opened a new log
I20260923 21:01:17.476184 8508 ts_tablet_manager.cc:1397] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260923 21:01:17.477114 8508 raft_consensus.cc:348] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.477372 8508 raft_consensus.cc:374] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.477478 8508 raft_consensus.cc:729] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.477743 8508 consensus_queue.cc:260] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.477932 8531 ts_tablet_manager.cc:1918] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:17.478127 8531 log.cc:1196] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/8a9fd19f86f84a8eafd38944fdf3e285
I20260923 21:01:17.478389 8508 ts_tablet_manager.cc:1428] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 21:01:17.478742 8531 ts_tablet_manager.cc:1939] T 8a9fd19f86f84a8eafd38944fdf3e285 P a997ec2c9ee4471fba5dc195544e2cba: Deleting consensus metadata
I20260923 21:01:17.479507 8507 tablet_bootstrap.cc:492] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: No bootstrap required, opened a new log
I20260923 21:01:17.479672 8507 ts_tablet_manager.cc:1397] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20260923 21:01:17.480767 8507 raft_consensus.cc:348] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.481022 8507 raft_consensus.cc:374] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.481123 8507 raft_consensus.cc:729] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a997ec2c9ee4471fba5dc195544e2cba, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.481967 8509 tablet_bootstrap.cc:492] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: Bootstrap starting.
I20260923 21:01:17.483546 8507 consensus_queue.cc:260] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.483805 8509 tablet_bootstrap.cc:654] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.484735 7984 catalog_manager.cc:4841] TS a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429): tablet 8a9fd19f86f84a8eafd38944fdf3e285 (table test-table [id=fc9d38c33eca4b4bbaefcbdec9ef6898]) successfully deleted
I20260923 21:01:17.484897 8507 ts_tablet_manager.cc:1428] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260923 21:01:17.486632 8509 tablet_bootstrap.cc:492] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: No bootstrap required, opened a new log
I20260923 21:01:17.486871 8509 ts_tablet_manager.cc:1397] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 21:01:17.487756 8509 raft_consensus.cc:348] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.488090 8509 raft_consensus.cc:374] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.488224 8509 raft_consensus.cc:729] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22220f420436443caa4bab5f96c8a5ec, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.488502 8509 consensus_queue.cc:260] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.489074 8509 ts_tablet_manager.cc:1428] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 21:01:17.537405 8514 raft_consensus.cc:482] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:17.537678 8514 raft_consensus.cc:504] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.539755 8514 leader_election.cc:288] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:17.547878 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.540671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35683 (local address 127.0.97.2:32775)
0923 21:01:17.540863 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.540870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.541430 (+ 560us) server_negotiation.cc:407] Connection header received
0923 21:01:17.541681 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.541687 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.541776 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.541884 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:17.542553 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.543624 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.545749 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.546776 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.546921 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.547025 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.547415 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.547562 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:01:17.548030 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.540258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32775 (local address 127.0.97.3:35683)
0923 21:01:17.541339 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.541357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.541511 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.541937 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.541943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.541976 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:17.542365 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.542381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.543765 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.543770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.545555 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.545571 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.546981 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.546988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.547198 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.547253 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.547371 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.547547 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":1,"thread_start_us":805,"threads_started":1}
I20260923 21:01:17.548854 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02c58cf3597414196c4f1b12ae3c4fa" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec" is_pre_election: true
I20260923 21:01:17.549103 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.539957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34429 (local address 127.0.97.3:45693)
0923 21:01:17.541034 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.541061 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.541275 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.541681 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.541686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.541710 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:17.541975 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.541989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.543674 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.543681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.546198 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.546218 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.548095 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.548101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.548359 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.548399 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.548569 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.548713 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":75,"thread_start_us":810,"threads_started":1}
I20260923 21:01:17.549141 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.540309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45693 (local address 127.0.97.1:34429)
0923 21:01:17.540593 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.540599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.541162 (+ 563us) server_negotiation.cc:407] Connection header received
0923 21:01:17.541454 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.541460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.541553 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.541671 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:17.542195 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.543495 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.546438 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.547642 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.547787 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.547907 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.548627 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.548800 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260923 21:01:17.549140 8298 raft_consensus.cc:2454] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 0.
I20260923 21:01:17.549670 8364 leader_election.cc:302] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:17.550174 8514 raft_consensus.cc:2781] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:17.550303 8514 raft_consensus.cc:482] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:17.550228 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02c58cf3597414196c4f1b12ae3c4fa" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a997ec2c9ee4471fba5dc195544e2cba" is_pre_election: true
I20260923 21:01:17.550736 8514 raft_consensus.cc:3037] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.550853 8154 raft_consensus.cc:2454] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 0.
I20260923 21:01:17.553363 8514 raft_consensus.cc:504] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.554069 8514 leader_election.cc:288] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 election: Requested vote from peers a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:17.554629 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02c58cf3597414196c4f1b12ae3c4fa" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a997ec2c9ee4471fba5dc195544e2cba"
I20260923 21:01:17.554683 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02c58cf3597414196c4f1b12ae3c4fa" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec"
I20260923 21:01:17.554965 8298 raft_consensus.cc:3037] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.554961 8154 raft_consensus.cc:3037] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.557672 8298 raft_consensus.cc:2454] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 1.
I20260923 21:01:17.558193 8364 leader_election.cc:302] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:17.558555 8514 raft_consensus.cc:2781] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:17.558794 8154 raft_consensus.cc:2454] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 1.
I20260923 21:01:17.559141 8514 raft_consensus.cc:686] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 1 LEADER]: Becoming Leader. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Running, Role: LEADER
I20260923 21:01:17.559829 8514 consensus_queue.cc:237] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.564549 7997 catalog_manager.cc:5495] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 reported cstate change: term changed from 0 to 1, leader changed from <none> to d37c7cd1ec92414986ab789cea413034 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } health_report { overall_health: HEALTHY } } }
Sep 23 21:01:17 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197275, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:17.620031 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.604844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33341 (local address 127.0.0.1:48102)
0923 21:01:17.605660 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.605683 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.605882 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.606309 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.606314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.607109 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:17.607354 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.607364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.608482 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.608490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.609430 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.609441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.610191 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.610197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.610451 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.611257 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:17.611275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:17.613775 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:17.617123 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:17.617141 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:17.617150 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:17.617671 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:17.618101 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:17.618107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:17.618111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:17.618253 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:17.618856 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:17.618865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:17.619037 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.619202 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.619567 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":26,"thread_start_us":413,"threads_started":1}
I20260923 21:01:17.620076 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.604936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.97.62:33341)
0923 21:01:17.606039 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.606043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.606063 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:01:17.606103 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.606106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.606162 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.606277 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:17.607562 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.608346 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.609578 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.609998 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.610139 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.614023 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:17.614028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:17.614032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:17.614062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:17.616964 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:17.617811 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:17.617826 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:17.617831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:17.617986 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:17.618411 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:17.618417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:17.618420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:17.618703 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:17.618866 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.619293 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.619698 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":956,"thread_start_us":757,"threads_started":1}
I20260923 21:01:17.626875 7997 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48102:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 21:01:17.628026 7997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:17.635396 8133 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc77c2c59e846a7ab43e49c476c9e56 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=df97ca29b56b4eb28aa6171bfa1eeb95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:01:17.635679 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc77c2c59e846a7ab43e49c476c9e56 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=df97ca29b56b4eb28aa6171bfa1eeb95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:01:17.636049 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc77c2c59e846a7ab43e49c476c9e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.636164 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc77c2c59e846a7ab43e49c476c9e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.635970 8273 tablet_service.cc:1467] Processing CreateTablet for tablet 0cc77c2c59e846a7ab43e49c476c9e56 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=df97ca29b56b4eb28aa6171bfa1eeb95]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 21:01:17.636662 8273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cc77c2c59e846a7ab43e49c476c9e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:17.643498 8508 tablet_bootstrap.cc:492] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: Bootstrap starting.
I20260923 21:01:17.644625 8509 tablet_bootstrap.cc:492] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: Bootstrap starting.
I20260923 21:01:17.645364 8508 tablet_bootstrap.cc:654] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.645494 8507 tablet_bootstrap.cc:492] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: Bootstrap starting.
I20260923 21:01:17.646548 8509 tablet_bootstrap.cc:654] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.647652 8507 tablet_bootstrap.cc:654] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:17.657126 8507 tablet_bootstrap.cc:492] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: No bootstrap required, opened a new log
I20260923 21:01:17.657897 8508 tablet_bootstrap.cc:492] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: No bootstrap required, opened a new log
I20260923 21:01:17.658102 8508 ts_tablet_manager.cc:1397] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.006s
I20260923 21:01:17.657349 8507 ts_tablet_manager.cc:1397] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260923 21:01:17.658622 8509 tablet_bootstrap.cc:492] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: No bootstrap required, opened a new log
I20260923 21:01:17.658927 8509 ts_tablet_manager.cc:1397] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260923 21:01:17.659075 8508 raft_consensus.cc:348] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.659376 8508 raft_consensus.cc:374] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.659467 8508 raft_consensus.cc:729] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.659353 8507 raft_consensus.cc:348] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.659606 8507 raft_consensus.cc:374] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.659688 8507 raft_consensus.cc:729] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a997ec2c9ee4471fba5dc195544e2cba, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.659863 8508 consensus_queue.cc:260] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.659998 8507 consensus_queue.cc:260] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.660321 8514 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.660611 8514 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:01:17.660393 8509 raft_consensus.cc:348] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.660502 8528 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.660880 8528 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:01:17.660979 8509 raft_consensus.cc:374] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:17.661106 8509 raft_consensus.cc:729] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22220f420436443caa4bab5f96c8a5ec, State: Initialized, Role: FOLLOWER
I20260923 21:01:17.661552 8509 consensus_queue.cc:260] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.662055 8515 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.662406 8515 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:01:17.662756 8508 ts_tablet_manager.cc:1428] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260923 21:01:17.663348 8509 ts_tablet_manager.cc:1428] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 21:01:17.664747 8507 ts_tablet_manager.cc:1428] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20260923 21:01:17.676342 8344 tablet.cc:2369] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:17.841799 8514 raft_consensus.cc:482] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:17.842082 8514 raft_consensus.cc:504] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.842769 8514 leader_election.cc:288] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:17.843343 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc77c2c59e846a7ab43e49c476c9e56" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec" is_pre_election: true
I20260923 21:01:17.843384 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc77c2c59e846a7ab43e49c476c9e56" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a997ec2c9ee4471fba5dc195544e2cba" is_pre_election: true
I20260923 21:01:17.843617 8298 raft_consensus.cc:2454] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 0.
I20260923 21:01:17.843667 8154 raft_consensus.cc:2454] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 0.
I20260923 21:01:17.844097 8364 leader_election.cc:302] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:17.844441 8514 raft_consensus.cc:2781] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:17.844554 8514 raft_consensus.cc:482] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:17.844663 8514 raft_consensus.cc:3037] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.846939 8514 raft_consensus.cc:504] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.847590 8514 leader_election.cc:288] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 election: Requested vote from peers a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:17.848203 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc77c2c59e846a7ab43e49c476c9e56" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec"
I20260923 21:01:17.848207 8154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cc77c2c59e846a7ab43e49c476c9e56" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a997ec2c9ee4471fba5dc195544e2cba"
I20260923 21:01:17.848464 8298 raft_consensus.cc:3037] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.848474 8154 raft_consensus.cc:3037] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:17.851610 8298 raft_consensus.cc:2454] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 1.
I20260923 21:01:17.851786 8154 raft_consensus.cc:2454] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 1.
I20260923 21:01:17.852162 8364 leader_election.cc:302] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:17.852519 8514 raft_consensus.cc:2781] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:17.852733 8514 raft_consensus.cc:686] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [term 1 LEADER]: Becoming Leader. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Running, Role: LEADER
I20260923 21:01:17.853015 8514 consensus_queue.cc:237] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:17.854749 8539 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: TxnStatusTablet state changed. Reason: New leader d37c7cd1ec92414986ab789cea413034. Latest consensus state: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.855065 8539 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:01:17.855789 8552 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 21:01:17.856030 8552 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 21:01:17.856503 7995 catalog_manager.cc:5495] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 reported cstate change: term changed from 0 to 1, leader changed from <none> to d37c7cd1ec92414986ab789cea413034 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } health_report { overall_health: HEALTHY } } }
I20260923 21:01:17.907689 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.899046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.0.1:33970)
0923 21:01:17.899462 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.899484 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.899678 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.900415 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.900420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.900435 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:01:17.900647 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.900656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.902432 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.902439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.904114 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.904137 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.905373 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.905381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:17.905599 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.905693 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:01:17.907049 (+ 1356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:01:17.907076 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 21:01:17.907182 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 21:01:17.907366 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260923 21:01:17.907783 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.898620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.0.97.3:36127)
0923 21:01:17.899321 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.899327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.899957 (+ 630us) server_negotiation.cc:407] Connection header received
0923 21:01:17.900073 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.900078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.900181 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.900354 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:17.900837 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.902218 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.904344 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.905150 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.905322 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.905994 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:17.906915 (+ 921us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:17.907079 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.907254 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.907352 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":1221,"threads_started":1}
I20260923 21:01:17.912956 8298 raft_consensus.cc:1261] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Refusing update from remote peer d37c7cd1ec92414986ab789cea413034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:17.913321 8154 raft_consensus.cc:1261] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Refusing update from remote peer d37c7cd1ec92414986ab789cea413034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:17.913842 8514 consensus_queue.cc:1035] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:17.914376 8539 consensus_queue.cc:1035] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:17.933314 8515 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: TxnStatusTablet state changed. Reason: New leader d37c7cd1ec92414986ab789cea413034. Latest consensus state: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.933634 8515 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:01:17.938319 8528 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: TxnStatusTablet state changed. Reason: New leader d37c7cd1ec92414986ab789cea413034. Latest consensus state: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.938686 8528 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:01:17.953397 8515 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.953732 8515 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P 22220f420436443caa4bab5f96c8a5ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:01:17.959091 8514 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.959547 8514 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:01:17.963052 8539 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.963356 8539 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P d37c7cd1ec92414986ab789cea413034: This TxnStatusTablet replica's current role is: LEADER
I20260923 21:01:17.964709 8528 tablet_replica.cc:439] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } } }
I20260923 21:01:17.964956 8528 tablet_replica.cc:442] T 0cc77c2c59e846a7ab43e49c476c9e56 P a997ec2c9ee4471fba5dc195544e2cba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 21:01:17.984819 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.971194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.97.3:36127)
0923 21:01:17.971425 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.971432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.979045 (+ 7613us) server_negotiation.cc:407] Connection header received
0923 21:01:17.979281 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.979286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.979376 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.979479 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:17.979953 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.981198 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.982770 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:17.983503 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:17.983630 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:17.984019 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:17.984162 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:17.984258 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:17.984421 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 21:01:17.984505 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 21:01:18.005434 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.997564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54707 (local address 127.0.97.3:36127)
0923 21:01:17.997807 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:17.997813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:17.998456 (+ 643us) server_negotiation.cc:407] Connection header received
0923 21:01:17.998674 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:17.998679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:17.998797 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:17.998898 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:17.999493 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.000710 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.003125 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.003851 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.004002 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.004556 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.004700 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.004801 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.004962 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.005067 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 21:01:18.005537 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:17.997630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.97.3:54707)
0923 21:01:17.998305 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:17.998353 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:17.998535 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:17.998929 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:17.998934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:17.998946 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:01:17.999323 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:17.999333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.000854 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.000860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:18.002942 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:18.002958 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.004065 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.004071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:18.004272 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.004345 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.004825 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.004836 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 21:01:18.004924 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 21:01:18.005090 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":355,"threads_started":1}
I20260923 21:01:18.016532 8152 raft_consensus.cc:1261] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Refusing update from remote peer d37c7cd1ec92414986ab789cea413034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:18.017594 8514 consensus_queue.cc:1035] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:18.018077 8298 raft_consensus.cc:1261] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Refusing update from remote peer d37c7cd1ec92414986ab789cea413034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:18.018868 8539 consensus_queue.cc:1035] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:18.033182 8523 mvcc.cc:205] Tried to move back new op lower bound from 7332648050747797504 to 7332648048889556992. Current Snapshot: MvccSnapshot[applied={T|T < 7332648050747797504}]
I20260923 21:01:18.101953 7995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:18.102185 7995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:18.105273 7995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Sending DeleteTablet for 3 replicas of tablet c02c58cf3597414196c4f1b12ae3c4fa
I20260923 21:01:18.106158 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet c02c58cf3597414196c4f1b12ae3c4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51628
I20260923 21:01:18.106293 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet c02c58cf3597414196c4f1b12ae3c4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260923 21:01:18.106248 8273 tablet_service.cc:1514] Processing DeleteTablet for tablet c02c58cf3597414196c4f1b12ae3c4fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 21:01:18.116688 8575 tablet_replica.cc:330] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: stopping tablet replica
I20260923 21:01:18.116981 8575 raft_consensus.cc:2229] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:18.118505 8575 raft_consensus.cc:2258] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.119942 8575 ts_tablet_manager.cc:1905] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.125952 8576 tablet_replica.cc:330] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: stopping tablet replica
I20260923 21:01:18.126408 8576 raft_consensus.cc:2229] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:01:18.127234 8576 raft_consensus.cc:2258] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.128640 8576 ts_tablet_manager.cc:1905] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.134967 8574 tablet_replica.cc:330] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: stopping tablet replica
I20260923 21:01:18.135224 8574 raft_consensus.cc:2229] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:18.135466 8574 raft_consensus.cc:2258] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.136860 8574 ts_tablet_manager.cc:1905] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.137338 8576 ts_tablet_manager.cc:1918] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:01:18.137507 8576 log.cc:1196] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/c02c58cf3597414196c4f1b12ae3c4fa
I20260923 21:01:18.138006 8576 ts_tablet_manager.cc:1939] T c02c58cf3597414196c4f1b12ae3c4fa P d37c7cd1ec92414986ab789cea413034: Deleting consensus metadata
I20260923 21:01:18.139247 7981 catalog_manager.cc:4841] TS d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127): tablet c02c58cf3597414196c4f1b12ae3c4fa (table test-table [id=aabd5a326e7e429ab08079f51abb7bd2]) successfully deleted
I20260923 21:01:18.140329 8575 ts_tablet_manager.cc:1918] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:01:18.140493 8575 log.cc:1196] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/c02c58cf3597414196c4f1b12ae3c4fa
I20260923 21:01:18.140990 8575 ts_tablet_manager.cc:1939] T c02c58cf3597414196c4f1b12ae3c4fa P 22220f420436443caa4bab5f96c8a5ec: Deleting consensus metadata
I20260923 21:01:18.142230 7982 catalog_manager.cc:4841] TS 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775): tablet c02c58cf3597414196c4f1b12ae3c4fa (table test-table [id=aabd5a326e7e429ab08079f51abb7bd2]) successfully deleted
I20260923 21:01:18.144737 8574 ts_tablet_manager.cc:1918] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:01:18.144894 8574 log.cc:1196] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/c02c58cf3597414196c4f1b12ae3c4fa
I20260923 21:01:18.145388 8574 ts_tablet_manager.cc:1939] T c02c58cf3597414196c4f1b12ae3c4fa P a997ec2c9ee4471fba5dc195544e2cba: Deleting consensus metadata
I20260923 21:01:18.146631 7984 catalog_manager.cc:4841] TS a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429): tablet c02c58cf3597414196c4f1b12ae3c4fa (table test-table [id=aabd5a326e7e429ab08079f51abb7bd2]) successfully deleted
Sep 23 21:01:18 dist-test-slave-8xn5 krb5kdc[7952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 21:01:18 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:18 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 21:01:18.202528 8583 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:18.176316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.0.97.62:33341)
0923 21:01:18.191906 (+ 15590us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.191916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.192515 (+ 599us) server_negotiation.cc:407] Connection header received
0923 21:01:18.192742 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.192748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.192836 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.192960 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:18.194393 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.195258 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.196559 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.197015 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.197157 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.202189 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48118: BlockingRecv error: recv got EOF from 127.0.0.1:48118 (error 108)
Metrics: {"server-negotiator.queue_time_us":15361,"thread_start_us":16079,"threads_started":1}
Sep 23 21:01:18 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260923 21:01:18.225613 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.206544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.0.97.62:33341)
0923 21:01:18.206759 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.206769 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.207180 (+ 411us) server_negotiation.cc:407] Connection header received
0923 21:01:18.207356 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.207360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.207423 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.207510 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:18.209361 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.210677 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.212669 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.213939 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.214148 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.219665 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:18.219671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:18.219674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:18.219716 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:18.222784 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:18.223517 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:18.223524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:18.223527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:18.223675 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:18.224097 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:18.224102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:18.224106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:18.224401 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:18.224669 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.225023 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.225288 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:01:18.229614 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:01:18.230279 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:18.237625 8133 tablet_service.cc:1467] Processing CreateTablet for tablet 16b773b35249499bad7f615948e83e40 (DEFAULT_TABLE table=test-table [id=073ec28eaa194e9fa9d3563fcc1d02b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:18.237888 8273 tablet_service.cc:1467] Processing CreateTablet for tablet 16b773b35249499bad7f615948e83e40 (DEFAULT_TABLE table=test-table [id=073ec28eaa194e9fa9d3563fcc1d02b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:18.238209 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b773b35249499bad7f615948e83e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:18.238263 8273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b773b35249499bad7f615948e83e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:18.242842 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 16b773b35249499bad7f615948e83e40 (DEFAULT_TABLE table=test-table [id=073ec28eaa194e9fa9d3563fcc1d02b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:18.243407 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b773b35249499bad7f615948e83e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:18.254171 8591 tablet_bootstrap.cc:492] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: Bootstrap starting.
I20260923 21:01:18.257229 8592 tablet_bootstrap.cc:492] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: Bootstrap starting.
I20260923 21:01:18.258062 8591 tablet_bootstrap.cc:654] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:18.260126 8592 tablet_bootstrap.cc:654] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:18.262773 8590 tablet_bootstrap.cc:492] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: Bootstrap starting.
I20260923 21:01:18.264634 8592 tablet_bootstrap.cc:492] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: No bootstrap required, opened a new log
I20260923 21:01:18.264689 8590 tablet_bootstrap.cc:654] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:18.264915 8592 ts_tablet_manager.cc:1397] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 21:01:18.266032 8592 raft_consensus.cc:348] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.266577 8592 raft_consensus.cc:374] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:18.266764 8592 raft_consensus.cc:729] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Initialized, Role: FOLLOWER
I20260923 21:01:18.267308 8592 consensus_queue.cc:260] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.271862 8592 ts_tablet_manager.cc:1428] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260923 21:01:18.273180 8590 tablet_bootstrap.cc:492] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: No bootstrap required, opened a new log
I20260923 21:01:18.273353 8590 ts_tablet_manager.cc:1397] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260923 21:01:18.274230 8590 raft_consensus.cc:348] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.274541 8590 raft_consensus.cc:374] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:18.274650 8590 raft_consensus.cc:729] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22220f420436443caa4bab5f96c8a5ec, State: Initialized, Role: FOLLOWER
I20260923 21:01:18.274909 8590 consensus_queue.cc:260] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.275483 8590 ts_tablet_manager.cc:1428] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 21:01:18.276026 8591 tablet_bootstrap.cc:492] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: No bootstrap required, opened a new log
I20260923 21:01:18.276217 8591 ts_tablet_manager.cc:1397] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.007s
I20260923 21:01:18.277053 8591 raft_consensus.cc:348] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.277323 8591 raft_consensus.cc:374] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:18.277431 8591 raft_consensus.cc:729] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a997ec2c9ee4471fba5dc195544e2cba, State: Initialized, Role: FOLLOWER
I20260923 21:01:18.277743 8591 consensus_queue.cc:260] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.278512 8591 ts_tablet_manager.cc:1428] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 21:01:18.326681 8572 raft_consensus.cc:482] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:18.326946 8572 raft_consensus.cc:504] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.327790 8572 leader_election.cc:288] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:18.328451 8152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b773b35249499bad7f615948e83e40" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a997ec2c9ee4471fba5dc195544e2cba" is_pre_election: true
I20260923 21:01:18.328433 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b773b35249499bad7f615948e83e40" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec" is_pre_election: true
I20260923 21:01:18.328735 8152 raft_consensus.cc:2454] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 0.
I20260923 21:01:18.328715 8298 raft_consensus.cc:2454] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 0.
I20260923 21:01:18.329264 8366 leader_election.cc:302] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a997ec2c9ee4471fba5dc195544e2cba, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:18.329655 8572 raft_consensus.cc:2781] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:18.329830 8572 raft_consensus.cc:482] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:18.329965 8572 raft_consensus.cc:3037] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:18.333899 8572 raft_consensus.cc:504] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.334878 8572 leader_election.cc:288] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 election: Requested vote from peers a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429), 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775)
I20260923 21:01:18.335511 8152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b773b35249499bad7f615948e83e40" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a997ec2c9ee4471fba5dc195544e2cba"
I20260923 21:01:18.335623 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b773b35249499bad7f615948e83e40" candidate_uuid: "d37c7cd1ec92414986ab789cea413034" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec"
I20260923 21:01:18.335813 8152 raft_consensus.cc:3037] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:18.335916 8298 raft_consensus.cc:3037] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:18.338464 8298 raft_consensus.cc:2454] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 1.
I20260923 21:01:18.339064 8364 leader_election.cc:302] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:18.339301 8152 raft_consensus.cc:2454] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d37c7cd1ec92414986ab789cea413034 in term 1.
I20260923 21:01:18.339486 8572 raft_consensus.cc:2781] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:18.339701 8572 raft_consensus.cc:686] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 1 LEADER]: Becoming Leader. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Running, Role: LEADER
I20260923 21:01:18.340008 8572 consensus_queue.cc:237] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.343565 7995 catalog_manager.cc:5495] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 reported cstate change: term changed from 0 to 1, leader changed from <none> to d37c7cd1ec92414986ab789cea413034 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d37c7cd1ec92414986ab789cea413034" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } health_report { overall_health: HEALTHY } } }
I20260923 21:01:18.363245 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.357012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33994 (local address 127.0.97.3:36127)
0923 21:01:18.357263 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.357268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.357356 (+ 88us) server_negotiation.cc:407] Connection header received
0923 21:01:18.357553 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.357556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.357620 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.357701 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:18.358205 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.359023 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.361273 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.361781 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.361883 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.362471 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.362597 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.362667 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.362877 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.362946 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 21:01:18.366143 8298 raft_consensus.cc:1261] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Refusing update from remote peer d37c7cd1ec92414986ab789cea413034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:18.366147 8152 raft_consensus.cc:1261] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Refusing update from remote peer d37c7cd1ec92414986ab789cea413034: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:18.366840 8514 consensus_queue.cc:1035] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:18.367153 8572 consensus_queue.cc:1035] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:18.382313 7995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:18.383242 7995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:18.387809 7995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Sending DeleteTablet for 3 replicas of tablet 16b773b35249499bad7f615948e83e40
I20260923 21:01:18.388984 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b773b35249499bad7f615948e83e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51628
I20260923 21:01:18.389304 8574 tablet_replica.cc:330] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: stopping tablet replica
I20260923 21:01:18.389505 8574 raft_consensus.cc:2229] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:18.389546 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b773b35249499bad7f615948e83e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260923 21:01:18.389567 8272 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b773b35249499bad7f615948e83e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 21:01:18.389750 8574 raft_consensus.cc:2258] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.389870 8273 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b773b35249499bad7f615948e83e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 21:01:18.389932 8576 tablet_replica.cc:330] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: stopping tablet replica
I20260923 21:01:18.390057 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b773b35249499bad7f615948e83e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260923 21:01:18.390146 8576 raft_consensus.cc:2229] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:01:18.390127 8575 tablet_replica.cc:330] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: stopping tablet replica
I20260923 21:01:18.390470 8575 raft_consensus.cc:2229] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:18.390533 8576 raft_consensus.cc:2258] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.390730 8575 raft_consensus.cc:2258] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.391779 8576 ts_tablet_manager.cc:1905] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.392932 8574 ts_tablet_manager.cc:1905] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.392750 7995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:01:18.393656 7995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:18.397670 8575 ts_tablet_manager.cc:1905] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.399623 8576 ts_tablet_manager.cc:1918] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:18.399796 8576 log.cc:1196] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/16b773b35249499bad7f615948e83e40
I20260923 21:01:18.400230 8576 ts_tablet_manager.cc:1939] T 16b773b35249499bad7f615948e83e40 P d37c7cd1ec92414986ab789cea413034: Deleting consensus metadata
I20260923 21:01:18.401360 7981 catalog_manager.cc:4841] TS d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127): tablet 16b773b35249499bad7f615948e83e40 (table test-table [id=073ec28eaa194e9fa9d3563fcc1d02b9]) successfully deleted
I20260923 21:01:18.401399 8574 ts_tablet_manager.cc:1918] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:18.401547 8574 log.cc:1196] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/16b773b35249499bad7f615948e83e40
I20260923 21:01:18.402033 8574 ts_tablet_manager.cc:1939] T 16b773b35249499bad7f615948e83e40 P a997ec2c9ee4471fba5dc195544e2cba: Deleting consensus metadata
I20260923 21:01:18.403690 7984 catalog_manager.cc:4841] TS a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429): tablet 16b773b35249499bad7f615948e83e40 (table test-table [id=073ec28eaa194e9fa9d3563fcc1d02b9]) successfully deleted
W20260923 21:01:18.405699 7981 catalog_manager.cc:4808] TS d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127): delete failed for tablet 16b773b35249499bad7f615948e83e40 because the tablet was not found. No further retry: Not found: Tablet not found: 16b773b35249499bad7f615948e83e40
I20260923 21:01:18.406497 8575 ts_tablet_manager.cc:1918] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:18.406787 8575 log.cc:1196] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/16b773b35249499bad7f615948e83e40
I20260923 21:01:18.407259 8575 ts_tablet_manager.cc:1939] T 16b773b35249499bad7f615948e83e40 P 22220f420436443caa4bab5f96c8a5ec: Deleting consensus metadata
I20260923 21:01:18.407506 8413 tablet_service.cc:1467] Processing CreateTablet for tablet a0b9e7aa877544acb0fcb767232bddd4 (DEFAULT_TABLE table=test-table [id=8a0858e8557b426f8b12d3f40e74d8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:18.407959 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b9e7aa877544acb0fcb767232bddd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:18.410764 8133 tablet_service.cc:1467] Processing CreateTablet for tablet a0b9e7aa877544acb0fcb767232bddd4 (DEFAULT_TABLE table=test-table [id=8a0858e8557b426f8b12d3f40e74d8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:18.411199 8133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b9e7aa877544acb0fcb767232bddd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:18.415179 8592 tablet_bootstrap.cc:492] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: Bootstrap starting.
I20260923 21:01:18.416739 8591 tablet_bootstrap.cc:492] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: Bootstrap starting.
I20260923 21:01:18.417094 8592 tablet_bootstrap.cc:654] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:18.417397 7982 catalog_manager.cc:4841] TS 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775): tablet 16b773b35249499bad7f615948e83e40 (table test-table [id=073ec28eaa194e9fa9d3563fcc1d02b9]) successfully deleted
W20260923 21:01:18.417706 7982 catalog_manager.cc:4808] TS 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775): delete failed for tablet 16b773b35249499bad7f615948e83e40 because the tablet was not found. No further retry: Not found: Tablet not found: 16b773b35249499bad7f615948e83e40
I20260923 21:01:18.418720 8591 tablet_bootstrap.cc:654] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:18.418882 8273 tablet_service.cc:1467] Processing CreateTablet for tablet a0b9e7aa877544acb0fcb767232bddd4 (DEFAULT_TABLE table=test-table [id=8a0858e8557b426f8b12d3f40e74d8dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:18.419467 8273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0b9e7aa877544acb0fcb767232bddd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:18.428826 8592 tablet_bootstrap.cc:492] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: No bootstrap required, opened a new log
I20260923 21:01:18.429126 8592 ts_tablet_manager.cc:1397] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260923 21:01:18.429669 8590 tablet_bootstrap.cc:492] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: Bootstrap starting.
I20260923 21:01:18.431829 8590 tablet_bootstrap.cc:654] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:18.431932 8592 raft_consensus.cc:348] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.432224 8592 raft_consensus.cc:374] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:18.432312 8592 raft_consensus.cc:729] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37c7cd1ec92414986ab789cea413034, State: Initialized, Role: FOLLOWER
I20260923 21:01:18.432595 8592 consensus_queue.cc:260] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.433284 8592 ts_tablet_manager.cc:1428] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20260923 21:01:18.434975 8590 tablet_bootstrap.cc:492] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: No bootstrap required, opened a new log
I20260923 21:01:18.435174 8590 ts_tablet_manager.cc:1397] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 21:01:18.436019 8590 raft_consensus.cc:348] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.436247 8590 raft_consensus.cc:374] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:18.436338 8590 raft_consensus.cc:729] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22220f420436443caa4bab5f96c8a5ec, State: Initialized, Role: FOLLOWER
I20260923 21:01:18.436595 8590 consensus_queue.cc:260] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.437260 8590 ts_tablet_manager.cc:1428] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 21:01:18.438395 8591 tablet_bootstrap.cc:492] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: No bootstrap required, opened a new log
I20260923 21:01:18.438575 8591 ts_tablet_manager.cc:1397] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20260923 21:01:18.439344 8591 raft_consensus.cc:348] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.439597 8591 raft_consensus.cc:374] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:18.439684 8591 raft_consensus.cc:729] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a997ec2c9ee4471fba5dc195544e2cba, State: Initialized, Role: FOLLOWER
I20260923 21:01:18.439963 8591 consensus_queue.cc:260] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.440582 8591 ts_tablet_manager.cc:1428] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 21:01:18.564564 8528 raft_consensus.cc:482] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:18.564826 8528 raft_consensus.cc:504] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.565585 8528 leader_election.cc:288] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775), d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127)
I20260923 21:01:18.566169 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9e7aa877544acb0fcb767232bddd4" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37c7cd1ec92414986ab789cea413034" is_pre_election: true
I20260923 21:01:18.566221 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9e7aa877544acb0fcb767232bddd4" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec" is_pre_election: true
I20260923 21:01:18.566522 8433 raft_consensus.cc:2454] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 0.
I20260923 21:01:18.566545 8298 raft_consensus.cc:2454] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 0.
I20260923 21:01:18.567037 8085 leader_election.cc:302] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22220f420436443caa4bab5f96c8a5ec, a997ec2c9ee4471fba5dc195544e2cba; no voters:
I20260923 21:01:18.567549 8528 raft_consensus.cc:2781] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:18.567677 8528 raft_consensus.cc:482] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:18.567778 8528 raft_consensus.cc:3037] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:18.569887 8528 raft_consensus.cc:504] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.570609 8528 leader_election.cc:288] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 election: Requested vote from peers 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775), d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127)
I20260923 21:01:18.571185 8433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9e7aa877544acb0fcb767232bddd4" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d37c7cd1ec92414986ab789cea413034"
I20260923 21:01:18.571180 8298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0b9e7aa877544acb0fcb767232bddd4" candidate_uuid: "a997ec2c9ee4471fba5dc195544e2cba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220f420436443caa4bab5f96c8a5ec"
I20260923 21:01:18.571405 8298 raft_consensus.cc:3037] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:18.571416 8433 raft_consensus.cc:3037] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:18.573608 8433 raft_consensus.cc:2454] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 1.
I20260923 21:01:18.574118 8084 leader_election.cc:302] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a997ec2c9ee4471fba5dc195544e2cba, d37c7cd1ec92414986ab789cea413034; no voters:
I20260923 21:01:18.574503 8528 raft_consensus.cc:2781] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:18.574517 8298 raft_consensus.cc:2454] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a997ec2c9ee4471fba5dc195544e2cba in term 1.
I20260923 21:01:18.580976 8528 raft_consensus.cc:686] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 1 LEADER]: Becoming Leader. State: Replica: a997ec2c9ee4471fba5dc195544e2cba, State: Running, Role: LEADER
I20260923 21:01:18.581370 8528 consensus_queue.cc:237] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } }
I20260923 21:01:18.584640 7995 catalog_manager.cc:5495] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba reported cstate change: term changed from 0 to 1, leader changed from <none> to a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a997ec2c9ee4471fba5dc195544e2cba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a997ec2c9ee4471fba5dc195544e2cba" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 } health_report { overall_health: UNKNOWN } } }
I20260923 21:01:18.609454 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.604099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.0.97.1:34429)
0923 21:01:18.604634 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.604642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.604662 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:01:18.604724 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.604727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.604785 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.604897 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:18.605381 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.606158 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.607582 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.608023 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.608164 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.608646 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.608771 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.608874 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.608993 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.609074 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":255,"threads_started":1}
I20260923 21:01:18.626300 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.620228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51715 (local address 127.0.97.3:36127)
0923 21:01:18.620429 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.620433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.621090 (+ 657us) server_negotiation.cc:407] Connection header received
0923 21:01:18.621328 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.621332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.621396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.621494 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:18.621966 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.622888 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.624335 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.624876 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.625016 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.625503 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.625614 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.625712 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.625857 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.625936 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 21:01:18.626430 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.620232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36127 (local address 127.0.97.1:51715)
0923 21:01:18.620899 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:18.620995 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:18.621179 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:18.621510 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:18.621514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:18.621526 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:01:18.621810 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:18.621819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.623038 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.623043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:18.624177 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:18.624190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.625080 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.625087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:18.625249 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.625315 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.625713 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.625726 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 21:01:18.625836 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 21:01:18.626001 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":360,"threads_started":1}
I20260923 21:01:18.637168 8433 raft_consensus.cc:1261] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Refusing update from remote peer a997ec2c9ee4471fba5dc195544e2cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:18.637496 8298 raft_consensus.cc:1261] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Refusing update from remote peer a997ec2c9ee4471fba5dc195544e2cba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:18.637900 8528 consensus_queue.cc:1035] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d37c7cd1ec92414986ab789cea413034" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:18.638250 8603 consensus_queue.cc:1035] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "22220f420436443caa4bab5f96c8a5ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:18.681114 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.674241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55161 (local address 127.0.97.1:34429)
0923 21:01:18.674450 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.674458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.674927 (+ 469us) server_negotiation.cc:407] Connection header received
0923 21:01:18.675126 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.675133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.675224 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.675315 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:18.675789 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.676847 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.679250 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.679669 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.679771 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.680348 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.680454 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.680523 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.680705 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.680812 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 21:01:18.681568 8614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.673965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34429 (local address 127.0.97.3:55161)
0923 21:01:18.674799 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:18.674819 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:18.674973 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:18.675343 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:18.675347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:18.675357 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 21:01:18.675623 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:18.675630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.677031 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.677036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:18.679079 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:18.679103 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.679870 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.679877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:18.680082 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.680143 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 21:01:18.680561 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 21:01:18.680573 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 21:01:18.680676 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 21:01:18.680853 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":482,"threads_started":1}
I20260923 21:01:18.733433 7997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:18.733741 7997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:18.737386 7997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e398c89678148f9b636c9bc43dadb29: Sending DeleteTablet for 3 replicas of tablet a0b9e7aa877544acb0fcb767232bddd4
I20260923 21:01:18.738566 8273 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b9e7aa877544acb0fcb767232bddd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55512
I20260923 21:01:18.738930 8575 tablet_replica.cc:330] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: stopping tablet replica
I20260923 21:01:18.738914 8133 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b9e7aa877544acb0fcb767232bddd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51628
I20260923 21:01:18.739112 8575 raft_consensus.cc:2229] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:18.739210 8574 tablet_replica.cc:330] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: stopping tablet replica
I20260923 21:01:18.739324 8575 raft_consensus.cc:2258] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.739372 8574 raft_consensus.cc:2229] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:01:18.739717 8574 raft_consensus.cc:2258] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.739704 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet a0b9e7aa877544acb0fcb767232bddd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33956
I20260923 21:01:18.740167 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 8068
I20260923 21:01:18.740216 8576 tablet_replica.cc:330] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: stopping tablet replica
I20260923 21:01:18.740423 8576 raft_consensus.cc:2229] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:18.740655 8576 raft_consensus.cc:2258] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:18.742274 8574 ts_tablet_manager.cc:1905] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.742415 8576 ts_tablet_manager.cc:1905] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.742810 8575 ts_tablet_manager.cc:1905] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:18.749684 8574 ts_tablet_manager.cc:1918] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:01:18.749668 8576 ts_tablet_manager.cc:1918] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:01:18.749881 8576 log.cc:1196] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/a0b9e7aa877544acb0fcb767232bddd4
I20260923 21:01:18.749925 8574 log.cc:1196] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/a0b9e7aa877544acb0fcb767232bddd4
I20260923 21:01:18.750384 8576 ts_tablet_manager.cc:1939] T a0b9e7aa877544acb0fcb767232bddd4 P d37c7cd1ec92414986ab789cea413034: Deleting consensus metadata
I20260923 21:01:18.750599 8574 ts_tablet_manager.cc:1939] T a0b9e7aa877544acb0fcb767232bddd4 P a997ec2c9ee4471fba5dc195544e2cba: Deleting consensus metadata
I20260923 21:01:18.751789 7981 catalog_manager.cc:4841] TS d37c7cd1ec92414986ab789cea413034 (127.0.97.3:36127): tablet a0b9e7aa877544acb0fcb767232bddd4 (table test-table [id=8a0858e8557b426f8b12d3f40e74d8dd]) successfully deleted
I20260923 21:01:18.752267 7984 catalog_manager.cc:4841] TS a997ec2c9ee4471fba5dc195544e2cba (127.0.97.1:34429): tablet a0b9e7aa877544acb0fcb767232bddd4 (table test-table [id=8a0858e8557b426f8b12d3f40e74d8dd]) successfully deleted
I20260923 21:01:18.753188 8575 ts_tablet_manager.cc:1918] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 21:01:18.753377 8575 log.cc:1196] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/a0b9e7aa877544acb0fcb767232bddd4
I20260923 21:01:18.753896 8575 ts_tablet_manager.cc:1939] T a0b9e7aa877544acb0fcb767232bddd4 P 22220f420436443caa4bab5f96c8a5ec: Deleting consensus metadata
I20260923 21:01:18.755189 7982 catalog_manager.cc:4841] TS 22220f420436443caa4bab5f96c8a5ec (127.0.97.2:32775): tablet a0b9e7aa877544acb0fcb767232bddd4 (table test-table [id=8a0858e8557b426f8b12d3f40e74d8dd]) successfully deleted
Sep 23 21:01:18 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260923 21:01:18.768322 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.752811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.0.97.1:34429)
0923 21:01:18.752998 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.753002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.753018 (+ 16us) server_negotiation.cc:407] Connection header received
0923 21:01:18.753066 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.753069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.753127 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.753261 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:18.754712 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.755600 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.756832 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.757254 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.757392 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.761652 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:18.761669 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:18.761686 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:18.761750 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:18.765043 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:18.765910 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:18.765918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:18.765921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:18.766071 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:18.766619 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:18.766623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:18.766626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:18.767020 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:18.767354 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.767709 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.767959 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:01:18.768977 8199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:18.913482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 8068
I20260923 21:01:18.937214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 8208
Sep 23 21:01:18 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260923 21:01:18.953428 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:18.938194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.0.97.2:32775)
0923 21:01:18.938732 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:18.938736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:18.938753 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:01:18.938807 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:18.938810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:18.938868 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:18.938965 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:18.940201 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.941077 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.942323 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:18.942827 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:18.942929 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:18.947355 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:18.947373 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:18.947389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:18.947439 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:18.950463 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:18.951300 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:18.951309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:18.951312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:18.951480 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:18.951830 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:18.951833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:18.951835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:18.952204 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:18.952523 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:18.952793 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 21:01:18.952983 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":333,"threads_started":1}
I20260923 21:01:18.954185 8338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:19.052855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 8208
I20260923 21:01:19.071168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 8347
Sep 23 21:01:19 dist-test-slave-8xn5 krb5kdc[7952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260923 21:01:19.087672 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:19.072151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.0.97.3:36127)
0923 21:01:19.072387 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:19.072391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:19.072407 (+ 16us) server_negotiation.cc:407] Connection header received
0923 21:01:19.072458 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:19.072462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:19.072520 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:19.072632 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:19.074007 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:19.074795 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:19.076065 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:19.076479 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:19.076620 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:19.081266 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:19.081289 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:19.081305 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:19.081349 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:19.084480 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:19.085386 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:19.085394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:19.085397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:19.085555 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:19.085962 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:19.085968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:19.085971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:19.086438 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:19.086746 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:19.087076 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 21:01:19.087331 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:01:19.088415 8478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:19.242578 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 8347
I20260923 21:01:19.268371 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 7965
I20260923 21:01:19.269632 8037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:19.391077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 7965
2026-09-23T21:01:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6074 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:01:19 dist-test-slave-8xn5 krb5kdc[8624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:01:19 dist-test-slave-8xn5 krb5kdc[8624](info): set up 2 sockets
Sep 23 21:01:19 dist-test-slave-8xn5 krb5kdc[8624](info): commencing operation
krb5kdc: starting...
W20260923 21:01:20.507630 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:01:20 dist-test-slave-8xn5 krb5kdc[8624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:01:20.747216 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43277
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:21.107683 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:21.108059 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:21.108242 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:21.115872 8637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:21.116055 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:21.116169 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:21.116273 8637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:21.116365 8637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:21.126914 8637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:21.128377 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:21.129792 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:21.142287 8642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:21.142419 8643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:21.142445 8645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:21.143527 8637 server_base.cc:1034] running on GCE node
I20260923 21:01:21.144377 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:21.145958 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:21.147190 8637 hybrid_clock.cc:648] HybridClock initialized: now 1790197281147122 us; error 68 us; skew 500 ppm
I20260923 21:01:21.147554 8637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T21:01:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1829 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:01:21 dist-test-slave-8xn5 krb5kdc[8651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:01:21 dist-test-slave-8xn5 krb5kdc[8651](info): set up 2 sockets
Sep 23 21:01:21 dist-test-slave-8xn5 krb5kdc[8651](info): commencing operation
krb5kdc: starting...
W20260923 21:01:22.347800 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:01:22 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:01:22.595311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40997
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:22.937608 8664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:22.937934 8664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:22.938090 8664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:22.945523 8664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:22.945660 8664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:22.945708 8664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:22.945824 8664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:22.945892 8664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:22.955969 8664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:22.957402 8664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:22.958874 8664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:22.972266 8669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:22.972301 8672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:22.972242 8670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:22.973603 8664 server_base.cc:1034] running on GCE node
I20260923 21:01:22.974870 8664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:22.977319 8664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:22.978691 8664 hybrid_clock.cc:648] HybridClock initialized: now 1790197282978575 us; error 118 us; skew 500 ppm
I20260923 21:01:22.979290 8664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:22 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:22.984030 8664 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:01:22.985956 8664 webserver.cc:460] Webserver started at http://127.0.97.62:43989/ using document root <none> and password file <none>
I20260923 21:01:22.986865 8664 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:22.987036 8664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:22.987425 8664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:22.991080 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "3ecadde4a0834fa8a85220205a95fde1"
format_stamp: "Formatted at 2026-09-23 21:01:22 on dist-test-slave-8xn5"
server_key: "751d8b990e9021d2a8504cc49ba0e275"
server_key_iv: "12baa37524e99563496673beb68ac41d"
server_key_version: "encryptionkey@0"
I20260923 21:01:22.992067 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "3ecadde4a0834fa8a85220205a95fde1"
format_stamp: "Formatted at 2026-09-23 21:01:22 on dist-test-slave-8xn5"
server_key: "751d8b990e9021d2a8504cc49ba0e275"
server_key_iv: "12baa37524e99563496673beb68ac41d"
server_key_version: "encryptionkey@0"
I20260923 21:01:22.998435 8664 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:01:23.002964 8679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:23.004653 8664 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:01:23.004890 8664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "3ecadde4a0834fa8a85220205a95fde1"
format_stamp: "Formatted at 2026-09-23 21:01:22 on dist-test-slave-8xn5"
server_key: "751d8b990e9021d2a8504cc49ba0e275"
server_key_iv: "12baa37524e99563496673beb68ac41d"
server_key_version: "encryptionkey@0"
I20260923 21:01:23.005133 8664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:23.044322 8664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:23.049564 8664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:23.049935 8664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:23.071679 8664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40997
I20260923 21:01:23.071717 8730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40997 every 8 connection(s)
I20260923 21:01:23.073285 8664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:01:23.078159 8731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:23.080511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 8664
I20260923 21:01:23.080771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:01:23.081502 388 external_mini_cluster.cc:1428] Setting key 5f37a1b324ba0bf8827a66eeb18ac85f
I20260923 21:01:23.092300 8731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Bootstrap starting.
I20260923 21:01:23.096798 8731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:23.098311 8731 log.cc:824] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:01:23 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:23.102896 8731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: No bootstrap required, opened a new log
I20260923 21:01:23.110594 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:23.084036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.0.97.62:40997)
0923 21:01:23.084965 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:23.084978 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:23.085023 (+ 45us) server_negotiation.cc:407] Connection header received
0923 21:01:23.086970 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:23.086996 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:23.087435 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:23.087877 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:23.089920 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:23.091428 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:23.093393 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:23.094186 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:23.094401 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:23.100107 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:23.100121 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:23.100255 (+ 134us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:23.100304 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:23.104934 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:23.105809 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:23.105819 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:23.105829 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:23.106080 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:23.106683 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:23.106695 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:23.106699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:23.107551 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:23.107889 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:23.108296 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 21:01:23.108814 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":436,"threads_started":1}
I20260923 21:01:23.114781 8731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:23.115204 8731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:23.115281 8731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ecadde4a0834fa8a85220205a95fde1, State: Initialized, Role: FOLLOWER
I20260923 21:01:23.115839 8731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:23.116061 8731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:01:23.116168 8731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:01:23.116302 8731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:23.118808 8731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:23.119376 8731 leader_election.cc:302] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ecadde4a0834fa8a85220205a95fde1; no voters:
I20260923 21:01:23.120055 8731 leader_election.cc:288] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:01:23.120236 8736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:23.121608 8736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 LEADER]: Becoming Leader. State: Replica: 3ecadde4a0834fa8a85220205a95fde1, State: Running, Role: LEADER
I20260923 21:01:23.122221 8736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:23.122663 8731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:23.125296 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ecadde4a0834fa8a85220205a95fde1. Latest consensus state: current_term: 1 leader_uuid: "3ecadde4a0834fa8a85220205a95fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } } }
I20260923 21:01:23.125404 8737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ecadde4a0834fa8a85220205a95fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } } }
I20260923 21:01:23.125679 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:23.125806 8737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:23.126713 8743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:23.134020 8743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:23.147374 8743 catalog_manager.cc:1347] Generated new cluster ID: 0708358070e7401682c7044188a69e1e
I20260923 21:01:23.147540 8743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:23.157864 8743 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:01:23.159549 8743 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:23.173097 8743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Generated new TSK 0
I20260923 21:01:23.174239 8743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:01:23.231719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40997
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:23.604547 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:23.604918 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:23.605093 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:23.613725 8757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:23.613930 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:23.614115 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:23.625173 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:23.626686 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:23.628101 8757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:23.644011 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:23.644066 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:23.644088 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:23.645830 8757 server_base.cc:1034] running on GCE node
I20260923 21:01:23.646672 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:23.647990 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:23.649272 8757 hybrid_clock.cc:648] HybridClock initialized: now 1790197283649221 us; error 48 us; skew 500 ppm
I20260923 21:01:23.649657 8757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:23 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:23.653347 8757 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:01:23.655103 8757 webserver.cc:460] Webserver started at http://127.0.97.1:40067/ using document root <none> and password file <none>
I20260923 21:01:23.655917 8757 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:23.656038 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:23.656360 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:23.659823 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "6cdd7f30b1ca4f87874a11c55a0bb49b"
format_stamp: "Formatted at 2026-09-23 21:01:23 on dist-test-slave-8xn5"
server_key: "4daa71c30b82ec9168eac53093e7aafd"
server_key_iv: "a9211a5da3890d15f4f0922d627b401c"
server_key_version: "encryptionkey@0"
I20260923 21:01:23.660745 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6cdd7f30b1ca4f87874a11c55a0bb49b"
format_stamp: "Formatted at 2026-09-23 21:01:23 on dist-test-slave-8xn5"
server_key: "4daa71c30b82ec9168eac53093e7aafd"
server_key_iv: "a9211a5da3890d15f4f0922d627b401c"
server_key_version: "encryptionkey@0"
I20260923 21:01:23.667098 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260923 21:01:23.671594 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:23.673460 8757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:01:23.673743 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "6cdd7f30b1ca4f87874a11c55a0bb49b"
format_stamp: "Formatted at 2026-09-23 21:01:23 on dist-test-slave-8xn5"
server_key: "4daa71c30b82ec9168eac53093e7aafd"
server_key_iv: "a9211a5da3890d15f4f0922d627b401c"
server_key_version: "encryptionkey@0"
I20260923 21:01:23.673982 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:23.709659 8757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:23.715072 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:23.715447 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:23.716571 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:23.718115 8757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:23.718246 8757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:23.718436 8757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:23.718514 8757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:23.755137 8757 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35567
I20260923 21:01:23.755239 8884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35567 every 8 connection(s)
I20260923 21:01:23.756505 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:23.760839 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 8757
I20260923 21:01:23.761062 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:01:23.761782 388 external_mini_cluster.cc:1428] Setting key 67805be921a8c6bb42c0ef1ab9cd80d7
Sep 23 21:01:23 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:23.780318 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:23.759448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48155 (local address 127.0.97.62:40997)
0923 21:01:23.759967 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:23.759974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:23.762426 (+ 2452us) server_negotiation.cc:407] Connection header received
0923 21:01:23.763460 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:23.763465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:23.763531 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:23.763706 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:23.765833 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:23.766673 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:23.768526 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:23.768952 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:23.769107 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:23.773280 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:23.773286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:23.773290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:23.773331 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:23.776360 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:23.777334 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:23.777340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:23.777344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:23.777495 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:23.778090 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:23.778096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:23.778099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:23.778485 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:23.778688 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:23.779762 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0923 21:01:23.779958 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
I20260923 21:01:23.782089 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:23.760003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:48155)
0923 21:01:23.760847 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:23.760891 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:23.763049 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:23.764130 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:23.764139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:23.764917 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:23.765613 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:23.765634 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:23.766851 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:23.766857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:23.768341 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:23.768357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:23.769224 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:23.769234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:23.769501 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:23.770278 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:23.770308 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:23.772987 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:23.776586 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:23.776600 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:23.776625 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:23.777163 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:23.777631 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:23.777638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:23.777642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:23.777945 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:23.778679 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:23.778691 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:23.779287 (+ 596us) client_negotiation.cc:769] Sending connection context
0923 21:01:23.779668 (+ 381us) client_negotiation.cc:240] Negotiation successful
0923 21:01:23.780374 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":376,"threads_started":1}
I20260923 21:01:23.784268 8885 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40997
I20260923 21:01:23.784672 8885 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:23.785754 8885 heartbeater.cc:511] Master 127.0.97.62:40997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:01:23.788691 8696 ts_manager.cc:194] Registered new tserver with Master: 6cdd7f30b1ca4f87874a11c55a0bb49b (127.0.97.1:35567)
I20260923 21:01:23.790565 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:01:23.822767 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40997
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:24.187788 8891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:24.188122 8891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:24.188302 8891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:24.196003 8891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:24.196146 8891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:24.196400 8891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:24.207258 8891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:24.208722 8891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:24.210853 8891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:24.229311 8899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:24.229242 8897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:24.229283 8896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:24.230088 8891 server_base.cc:1034] running on GCE node
I20260923 21:01:24.230863 8891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:24.232342 8891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:24.233642 8891 hybrid_clock.cc:648] HybridClock initialized: now 1790197284233570 us; error 53 us; skew 500 ppm
I20260923 21:01:24.234057 8891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:24 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:24.237871 8891 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:01:24.239473 8891 webserver.cc:460] Webserver started at http://127.0.97.2:36353/ using document root <none> and password file <none>
I20260923 21:01:24.240280 8891 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:24.240396 8891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:24.240789 8891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:24.244073 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "9f9b8cf0e213474faecb3a0fe948d1f1"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "9595aca8a110464f8739513873372e6b"
server_key_iv: "9fba7e780a6ee46012be55343a3a94ec"
server_key_version: "encryptionkey@0"
I20260923 21:01:24.245011 8891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9f9b8cf0e213474faecb3a0fe948d1f1"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "9595aca8a110464f8739513873372e6b"
server_key_iv: "9fba7e780a6ee46012be55343a3a94ec"
server_key_version: "encryptionkey@0"
I20260923 21:01:24.250941 8891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 21:01:24.255048 8906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:24.256717 8891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:01:24.256925 8891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "9f9b8cf0e213474faecb3a0fe948d1f1"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "9595aca8a110464f8739513873372e6b"
server_key_iv: "9fba7e780a6ee46012be55343a3a94ec"
server_key_version: "encryptionkey@0"
I20260923 21:01:24.257164 8891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:24.292342 8891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:24.297545 8891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:24.297925 8891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:24.299100 8891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:24.300639 8891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:24.300742 8891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:24.300886 8891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:24.300936 8891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:24.340505 8891 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42407
I20260923 21:01:24.340610 9018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42407 every 8 connection(s)
I20260923 21:01:24.341948 8891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:24.348219 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 8891
I20260923 21:01:24.348405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:01:24.348989 388 external_mini_cluster.cc:1428] Setting key bfbf86828b3a6c65ad137b12591d0441
Sep 23 21:01:24 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:24.369201 9021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:24.345036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59055 (local address 127.0.97.62:40997)
0923 21:01:24.345568 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:24.345573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:24.346585 (+ 1012us) server_negotiation.cc:407] Connection header received
0923 21:01:24.349306 (+ 2721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:24.349312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:24.349409 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:24.349572 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:24.352052 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.352819 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.355707 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.356146 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.356253 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:24.360961 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:24.360971 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:24.360980 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:24.361033 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:24.365160 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:24.366197 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:24.366206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:24.366212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:24.366413 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:24.366956 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:24.366964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:24.366970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:24.367368 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:24.367543 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:24.368388 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 21:01:24.368679 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":302,"threads_started":1}
I20260923 21:01:24.370303 9022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:24.345511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:59055)
0923 21:01:24.346374 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:24.346442 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:24.348869 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:24.349899 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:24.349916 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:24.350702 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:24.351818 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:24.351851 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.353132 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.353141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:24.355494 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:24.355515 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.356412 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.356421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:24.356778 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:24.357676 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:24.357720 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:24.360586 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:24.365360 (+ 4774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:24.365369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:24.365386 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:24.366030 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:24.366568 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:24.366578 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:24.366581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:24.366805 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:24.367540 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:24.367549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:24.367963 (+ 414us) client_negotiation.cc:769] Sending connection context
0923 21:01:24.368250 (+ 287us) client_negotiation.cc:240] Negotiation successful
0923 21:01:24.368901 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":333,"threads_started":1}
I20260923 21:01:24.372215 9019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40997
I20260923 21:01:24.372599 9019 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:24.373596 9019 heartbeater.cc:511] Master 127.0.97.62:40997 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:01:24.375479 8696 ts_manager.cc:194] Registered new tserver with Master: 9f9b8cf0e213474faecb3a0fe948d1f1 (127.0.97.2:42407)
I20260923 21:01:24.376408 8696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:01:24.410578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40997
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:24.754949 9025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:24.755362 9025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:24.755522 9025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:24.765491 9025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:24.765623 9025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:24.765744 9025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:24.776161 9025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:24.777608 9025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:24.779219 9025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:24.793659 9030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:24.793735 9031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:24.793715 9033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:24.794713 9025 server_base.cc:1034] running on GCE node
I20260923 21:01:24.795403 8885 heartbeater.cc:503] Master 127.0.97.62:40997 was elected leader, sending a full tablet report...
I20260923 21:01:24.795518 9025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:24.796746 9025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:24.798002 9025 hybrid_clock.cc:648] HybridClock initialized: now 1790197284797929 us; error 71 us; skew 500 ppm
I20260923 21:01:24.798455 9025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:24 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:24.801934 9025 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:01:24.803735 9025 webserver.cc:460] Webserver started at http://127.0.97.3:34795/ using document root <none> and password file <none>
I20260923 21:01:24.804523 9025 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:24.804684 9025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:24.805019 9025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:24.808347 9025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "7294c77e126c45dc8224bbbbbabe0c78"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "4983e59bc76bf01b18b4107bdabd1e30"
server_key_iv: "53b746a2ceab4fe30f2cdf8a41232625"
server_key_version: "encryptionkey@0"
I20260923 21:01:24.809278 9025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7294c77e126c45dc8224bbbbbabe0c78"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "4983e59bc76bf01b18b4107bdabd1e30"
server_key_iv: "53b746a2ceab4fe30f2cdf8a41232625"
server_key_version: "encryptionkey@0"
I20260923 21:01:24.815810 9025 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 21:01:24.819844 9040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:24.821692 9025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:01:24.821941 9025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "7294c77e126c45dc8224bbbbbabe0c78"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "4983e59bc76bf01b18b4107bdabd1e30"
server_key_iv: "53b746a2ceab4fe30f2cdf8a41232625"
server_key_version: "encryptionkey@0"
I20260923 21:01:24.822191 9025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:24.835631 9025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:24.840713 9025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:24.841084 9025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:24.842417 9025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:24.844023 9025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:24.844143 9025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:24.844312 9025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:24.844396 9025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:24.880949 9025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34301
I20260923 21:01:24.881001 9152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34301 every 8 connection(s)
I20260923 21:01:24.882902 9025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:24.884918 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9025
I20260923 21:01:24.885089 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:01:24.885622 388 external_mini_cluster.cc:1428] Setting key 63a9cfb1ed41da31329e3a51f097341a
Sep 23 21:01:24 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:24.908052 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:24.886665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57735 (local address 127.0.97.62:40997)
0923 21:01:24.890688 (+ 4023us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:24.890692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:24.890719 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:01:24.891394 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:24.891398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:24.891501 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:24.891606 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:24.893754 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.894760 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.896495 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.896941 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.897175 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:24.901514 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:24.901520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:24.901523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:24.901581 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:24.904388 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:24.905264 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:24.905268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:24.905270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:24.905398 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:24.906145 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:24.906149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:24.906151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:24.906506 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:24.906676 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:24.907468 (+ 792us) server_negotiation.cc:299] Negotiation successful
0923 21:01:24.907675 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3911,"thread_start_us":3820,"threads_started":1}
I20260923 21:01:24.908994 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:24.886783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:57735)
0923 21:01:24.888208 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:24.888289 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:24.891072 (+ 2783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:24.891929 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:24.891941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:24.892775 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:24.893539 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:24.893559 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.894926 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.894930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:24.896336 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:24.896353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:24.897271 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:24.897276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:24.897487 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:24.898213 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:24.898236 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:24.901226 (+ 2990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:24.904563 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:24.904572 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:24.904589 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:24.905067 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:24.905531 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:24.905535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:24.905538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:24.905766 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:24.906675 (+ 909us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:24.906684 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:24.907057 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 21:01:24.907341 (+ 284us) client_negotiation.cc:240] Negotiation successful
0923 21:01:24.907779 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":689,"threads_started":1}
I20260923 21:01:24.910748 9153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40997
I20260923 21:01:24.911203 9153 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:24.912066 9153 heartbeater.cc:511] Master 127.0.97.62:40997 requested a full tablet report, sending...
I20260923 21:01:24.913931 8695 ts_manager.cc:194] Registered new tserver with Master: 7294c77e126c45dc8224bbbbbabe0c78 (127.0.97.3:34301)
I20260923 21:01:24.915328 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:01:24.915295 8695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57735
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260923 21:01:25.074350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 8757
Sep 23 21:01:25 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:01:25.093060 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:25.075530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.97.1:35567)
0923 21:01:25.076255 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:25.076268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:25.076673 (+ 405us) server_negotiation.cc:407] Connection header received
0923 21:01:25.076899 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:25.076906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:25.077310 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:25.077501 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:25.079196 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:25.080356 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:25.081695 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:25.082308 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:25.082529 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:25.086913 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:25.086928 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:25.086944 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:25.086993 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:25.089997 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:25.090842 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:25.090859 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:25.090862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:25.090992 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:25.091412 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:25.091416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:25.091419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:25.091782 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:25.092114 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:25.092409 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 21:01:25.092689 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":351,"threads_started":1}
I20260923 21:01:25.094592 8880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:25.179991 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 8757
I20260923 21:01:25.197973 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 8891
Sep 23 21:01:25 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:01:25.218279 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:25.199143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.0.97.2:42407)
0923 21:01:25.199735 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:25.199742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:25.199770 (+ 28us) server_negotiation.cc:407] Connection header received
0923 21:01:25.199839 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:25.199845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:25.200062 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:25.200206 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:25.201604 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:25.203043 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:25.204668 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:25.205318 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:25.205446 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:25.211240 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:25.211256 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:25.211273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:25.211351 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:25.214789 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:25.215699 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:25.215709 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:25.215711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:25.215863 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:25.216391 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:25.216394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:25.216396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:25.216804 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:25.217022 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:25.217474 (+ 452us) server_negotiation.cc:299] Negotiation successful
0923 21:01:25.217793 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":339,"threads_started":1}
I20260923 21:01:25.219782 9014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:25.306281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 8891
I20260923 21:01:25.325510 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9025
Sep 23 21:01:25 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:01:25.345606 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:25.326894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.0.97.3:34301)
0923 21:01:25.327603 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:25.327615 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:25.327665 (+ 50us) server_negotiation.cc:407] Connection header received
0923 21:01:25.327736 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:25.327743 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:25.328101 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:25.328269 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:25.329991 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:25.331352 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:25.332889 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:25.333628 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:25.333764 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:25.338571 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:25.338603 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:25.338622 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:25.338670 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:25.342093 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:25.343293 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:25.343302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:25.343305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:25.343457 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:25.343881 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:25.343884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:25.343887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:25.344367 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:25.344612 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:25.344903 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 21:01:25.345168 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":324,"threads_started":1}
I20260923 21:01:25.347131 9148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:25.432717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9025
I20260923 21:01:25.448892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 8664
I20260923 21:01:25.450313 8726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:25.555320 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 8664
I20260923 21:01:25.576731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40997
--webserver_interface=127.0.97.62
--webserver_port=43989
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:25.921317 9171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:25.921643 9171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:25.921732 9171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:25.929100 9171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:25.929227 9171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:25.929275 9171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:25.929335 9171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:25.929380 9171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:25.939515 9171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:25.940896 9171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:25.942379 9171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:25.955400 9176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:25.955359 9179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:25.955334 9177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:25.956267 9171 server_base.cc:1034] running on GCE node
I20260923 21:01:25.957230 9171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:25.959043 9171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:25.960305 9171 hybrid_clock.cc:648] HybridClock initialized: now 1790197285960245 us; error 99 us; skew 500 ppm
I20260923 21:01:25.960733 9171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:25 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:25.972076 9171 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:01:25.973905 9171 webserver.cc:460] Webserver started at http://127.0.97.62:43989/ using document root <none> and password file <none>
I20260923 21:01:25.974797 9171 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:25.974931 9171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:25.980906 9171 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260923 21:01:25.984230 9186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:25.986109 9171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 21:01:25.986299 9171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "3ecadde4a0834fa8a85220205a95fde1"
format_stamp: "Formatted at 2026-09-23 21:01:22 on dist-test-slave-8xn5"
server_key: "751d8b990e9021d2a8504cc49ba0e275"
server_key_iv: "12baa37524e99563496673beb68ac41d"
server_key_version: "encryptionkey@0"
I20260923 21:01:25.987432 9171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:26.007228 9171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:26.013039 9171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:26.013324 9171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:26.033982 9171 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40997
I20260923 21:01:26.034024 9237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40997 every 8 connection(s)
I20260923 21:01:26.035630 9171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:01:26.041024 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9171
I20260923 21:01:26.041900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35567
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:40997
--webserver_port=40067
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:01:26.043342 9238 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:01:26.046983 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Bootstrap starting.
I20260923 21:01:26.075582 9238 log.cc:824] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:26.087705 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:01:26.088384 9238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Bootstrap complete.
I20260923 21:01:26.102383 9238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:26.103771 9238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ecadde4a0834fa8a85220205a95fde1, State: Initialized, Role: FOLLOWER
I20260923 21:01:26.104700 9238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:26.105029 9238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:01:26.105172 9238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:01:26.105365 9238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 1 FOLLOWER]: Advancing to term 2
I20260923 21:01:26.111661 9238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:26.112522 9238 leader_election.cc:302] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ecadde4a0834fa8a85220205a95fde1; no voters:
I20260923 21:01:26.113381 9238 leader_election.cc:288] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 21:01:26.113556 9242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 21:01:26.115161 9242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [term 2 LEADER]: Becoming Leader. State: Replica: 3ecadde4a0834fa8a85220205a95fde1, State: Running, Role: LEADER
I20260923 21:01:26.115931 9238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:26.115998 9242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } }
I20260923 21:01:26.118921 9244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ecadde4a0834fa8a85220205a95fde1. Latest consensus state: current_term: 2 leader_uuid: "3ecadde4a0834fa8a85220205a95fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } } }
I20260923 21:01:26.118921 9243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3ecadde4a0834fa8a85220205a95fde1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ecadde4a0834fa8a85220205a95fde1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40997 } } }
I20260923 21:01:26.119561 9243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:26.119562 9244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:26.131404 9248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:26.138847 9248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:26.140450 9248 catalog_manager.cc:1259] Loaded cluster ID: 0708358070e7401682c7044188a69e1e
I20260923 21:01:26.140682 9248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:26.144963 9248 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:26.147068 9248 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3ecadde4a0834fa8a85220205a95fde1: Loaded TSK: 0
I20260923 21:01:26.148823 9248 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 21:01:26.445240 9240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:26.445653 9240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:26.445828 9240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:26.454041 9240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:26.454264 9240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:26.454487 9240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:26.465302 9240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35567
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40067
--tserver_master_addrs=127.0.97.62:40997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:26.467052 9240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:26.468519 9240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:26.485558 9265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:26.486292 9264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:26.486817 9267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:26.487262 9240 server_base.cc:1034] running on GCE node
I20260923 21:01:26.487962 9240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:26.489194 9240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:26.490494 9240 hybrid_clock.cc:648] HybridClock initialized: now 1790197286490424 us; error 51 us; skew 500 ppm
I20260923 21:01:26.490923 9240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:26 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:26.494881 9240 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260923 21:01:26.496734 9240 webserver.cc:460] Webserver started at http://127.0.97.1:40067/ using document root <none> and password file <none>
I20260923 21:01:26.497591 9240 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:26.497845 9240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:26.504505 9240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:01:26.508263 9274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:26.510303 9240 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:01:26.510612 9240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "6cdd7f30b1ca4f87874a11c55a0bb49b"
format_stamp: "Formatted at 2026-09-23 21:01:23 on dist-test-slave-8xn5"
server_key: "4daa71c30b82ec9168eac53093e7aafd"
server_key_iv: "a9211a5da3890d15f4f0922d627b401c"
server_key_version: "encryptionkey@0"
I20260923 21:01:26.511718 9240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:26.566753 9240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:26.571787 9240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:26.572155 9240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:26.573305 9240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:26.574980 9240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:26.575081 9240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:26.575198 9240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:26.575302 9240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:26.612139 9240 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35567
I20260923 21:01:26.612339 9386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35567 every 8 connection(s)
I20260923 21:01:26.613682 9240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:26.623402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9240
I20260923 21:01:26.624063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42407
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:40997
--webserver_port=36353
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:01:26 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:26 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:26.636246 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:26.617409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:60567)
0923 21:01:26.618581 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:26.618629 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:26.620588 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:26.622701 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:26.622710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:26.623292 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:26.623917 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:26.623935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.626556 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.626561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:26.627980 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:26.627995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.629660 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.629665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:26.629870 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:26.630769 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:26.630810 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:26.635167 (+ 4357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":653,"threads_started":1}
W20260923 21:01:26.636844 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:26.617224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60567 (local address 127.0.97.62:40997)
0923 21:01:26.618023 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:26.618047 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:26.618801 (+ 754us) server_negotiation.cc:407] Connection header received
0923 21:01:26.621400 (+ 2599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:26.621417 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:26.621864 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:26.622522 (+ 658us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:26.624158 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.626292 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.628183 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.629385 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.629619 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:26.635640 (+ 6021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60567: BlockingRecv error: recv got EOF from 127.0.97.1:60567 (error 108)
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":301,"threads_started":1}
W20260923 21:01:26.637255 9387 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40997 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40997: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 23 21:01:26 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:26 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:26.648864 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:26.638248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:51833)
0923 21:01:26.638515 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:26.638539 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:26.638679 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:26.639015 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:26.639022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:26.639507 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:26.639729 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:26.639743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.640811 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.640819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:26.642613 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:26.642634 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.643575 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.643584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:26.643835 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:26.644519 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:26.644550 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:26.648498 (+ 3948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260923 21:01:26.648948 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:26.638369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51833 (local address 127.0.97.62:40997)
0923 21:01:26.638532 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:26.638536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:26.638620 (+ 84us) server_negotiation.cc:407] Connection header received
0923 21:01:26.638797 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:26.638801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:26.638864 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:26.638947 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:26.639879 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.640646 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.642820 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.643382 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.643514 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:26.648731 (+ 5217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51833: BlockingRecv error: recv got EOF from 127.0.97.1:51833 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 21:01:26 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:26 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:26.658422 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:26.649612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:40961)
0923 21:01:26.649862 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:26.649888 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:26.650036 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:26.650673 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:26.650680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:26.650992 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:26.651162 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:26.651171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.652279 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.652284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:26.653225 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:26.653237 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.653960 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.653966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:26.654142 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:26.654789 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:26.654808 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:26.658055 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":545,"threads_started":1}
W20260923 21:01:26.658640 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:26.649775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40961 (local address 127.0.97.62:40997)
0923 21:01:26.650014 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:26.650022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:26.650049 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:01:26.650253 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:26.650259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:26.650368 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:26.650447 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:26.651312 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.652130 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.653383 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:26.653801 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:26.653945 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:26.658284 (+ 4339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40961: BlockingRecv error: recv got EOF from 127.0.97.1:40961 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 21:01:26.658962 9387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 21:01:26.981168 9391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:26.981484 9391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:26.981580 9391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:26.989181 9391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:26.989342 9391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:26.989464 9391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:27.000144 9391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36353
--tserver_master_addrs=127.0.97.62:40997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:27.001713 9391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:27.003261 9391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:27.018426 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:27.018321 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:27.019042 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:27.019965 9391 server_base.cc:1034] running on GCE node
I20260923 21:01:27.020819 9391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:27.022046 9391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:27.023319 9391 hybrid_clock.cc:648] HybridClock initialized: now 1790197287023238 us; error 72 us; skew 500 ppm
I20260923 21:01:27.023697 9391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:27.027493 9391 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260923 21:01:27.029323 9391 webserver.cc:460] Webserver started at http://127.0.97.2:36353/ using document root <none> and password file <none>
I20260923 21:01:27.030249 9391 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:27.030498 9391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:27.036955 9391 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:01:27.040277 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:27.042030 9391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:01:27.042219 9391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "9f9b8cf0e213474faecb3a0fe948d1f1"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "9595aca8a110464f8739513873372e6b"
server_key_iv: "9fba7e780a6ee46012be55343a3a94ec"
server_key_version: "encryptionkey@0"
I20260923 21:01:27.043295 9391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:27.066149 9391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:27.070848 9391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:27.071097 9391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:27.072106 9391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:27.073546 9391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:27.073647 9391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:27.073725 9391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:27.073779 9391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:27.110488 9391 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42407
I20260923 21:01:27.110589 9519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42407 every 8 connection(s)
I20260923 21:01:27.112331 9391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:27.122785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9391
I20260923 21:01:27.123540 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34301
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:40997
--webserver_port=34795
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 21:01:27.132035 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:27.115237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43115 (local address 127.0.97.62:40997)
0923 21:01:27.115425 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.115430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.116944 (+ 1514us) server_negotiation.cc:407] Connection header received
0923 21:01:27.119154 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.119159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.119250 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.119353 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.121187 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.121936 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.124873 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.125316 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.125440 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.131771 (+ 6331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43115: BlockingRecv error: recv got EOF from 127.0.97.2:43115 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:01:27.132903 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.115856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:43115)
0923 21:01:27.116749 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:27.116816 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:27.118803 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:27.119629 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:27.119639 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:27.120225 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:27.120970 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.120991 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.122190 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.122195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.124640 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.124665 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.125626 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.125636 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.125997 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.126864 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:27.126906 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:27.131538 (+ 4632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":303,"threads_started":1}
W20260923 21:01:27.133819 9520 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40997 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40997: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:27.145062 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.134608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:47123)
0923 21:01:27.134816 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:27.134838 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:27.134970 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:27.135323 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:27.135330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:27.135811 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:27.136033 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.136047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.137081 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.137089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.138824 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.138845 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.139846 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.139856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.140113 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.140894 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:27.140924 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:27.144742 (+ 3818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 21:01:27.145263 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:27.134671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47123 (local address 127.0.97.62:40997)
0923 21:01:27.134858 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.134863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.134886 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:01:27.135102 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.135106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.135170 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.135262 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.136197 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.136916 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.139228 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.139660 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.139772 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.144985 (+ 5213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47123: BlockingRecv error: recv got EOF from 127.0.97.2:47123 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:27.154697 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.145959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:51641)
0923 21:01:27.146276 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:27.146293 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:27.146442 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:27.146860 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:27.146863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:27.147288 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:27.147450 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.147458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.148550 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.148558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.149518 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.149529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.150233 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.150239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.150444 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.151191 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:27.151209 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:27.154335 (+ 3126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201}
W20260923 21:01:27.154819 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:27.146136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51641 (local address 127.0.97.62:40997)
0923 21:01:27.146384 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.146393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.146419 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:01:27.146577 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.146584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.146694 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.146828 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.147636 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.148407 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.149665 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.150077 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.150219 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.154590 (+ 4371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51641: BlockingRecv error: recv got EOF from 127.0.97.2:51641 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 21:01:27.155179 9520 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 21:01:27.492930 9523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:27.493287 9523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:27.493461 9523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:27.501219 9523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:27.501400 9523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:27.501600 9523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:27.511952 9523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34795
--tserver_master_addrs=127.0.97.62:40997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:27.513633 9523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:27.515161 9523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:27.530606 9528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:27.530678 9529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:27.530547 9531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:27.532335 9523 server_base.cc:1034] running on GCE node
I20260923 21:01:27.533250 9523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:27.534510 9523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:27.535794 9523 hybrid_clock.cc:648] HybridClock initialized: now 1790197287535739 us; error 47 us; skew 500 ppm
I20260923 21:01:27.536216 9523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:27.540309 9523 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260923 21:01:27.542258 9523 webserver.cc:460] Webserver started at http://127.0.97.3:34795/ using document root <none> and password file <none>
I20260923 21:01:27.543222 9523 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:27.543411 9523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:27.550863 9523 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 21:01:27.554649 9538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:27.556792 9523 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:01:27.556993 9523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "7294c77e126c45dc8224bbbbbabe0c78"
format_stamp: "Formatted at 2026-09-23 21:01:24 on dist-test-slave-8xn5"
server_key: "4983e59bc76bf01b18b4107bdabd1e30"
server_key_iv: "53b746a2ceab4fe30f2cdf8a41232625"
server_key_version: "encryptionkey@0"
I20260923 21:01:27.558056 9523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:27.583491 9523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:27.588742 9523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:27.589025 9523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:27.590157 9523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:27.591894 9523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:27.591997 9523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:27.592098 9523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:27.592159 9523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:27.635726 9523 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34301
I20260923 21:01:27.635795 9650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34301 every 8 connection(s)
I20260923 21:01:27.637435 9523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:27.643011 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9523
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 21:01:27.658744 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:27.640711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60589 (local address 127.0.97.62:40997)
0923 21:01:27.640927 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.640932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.642269 (+ 1337us) server_negotiation.cc:407] Connection header received
0923 21:01:27.645186 (+ 2917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.645191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.645323 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.645458 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.647750 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.648983 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.651319 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.651961 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.652122 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.658398 (+ 6276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60589: BlockingRecv error: recv got EOF from 127.0.97.3:60589 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"spinlock_wait_cycles":32896}
I20260923 21:01:27.659806 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.641291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:60589)
0923 21:01:27.642082 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:27.642124 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:27.644846 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:27.645840 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:27.645851 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:27.646538 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:27.647413 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.647524 (+ 111us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.649153 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.649158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.651126 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.651146 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.652256 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.652262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.652522 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.653418 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:27.653446 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:27.658161 (+ 4715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":330,"threads_started":1}
W20260923 21:01:27.661078 9651 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:40997 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:40997: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260923 21:01:27.662519 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.644352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.97.62:40997)
0923 21:01:27.645136 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.645141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.645160 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:01:27.645210 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.645213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.645297 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.645401 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.647261 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.648785 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.650125 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.650874 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.651042 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.654200 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:27.654219 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:27.654249 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:27.654306 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:27.657902 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:27.658826 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:27.658839 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:27.658848 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:27.659341 (+ 493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:27.659808 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:27.659815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:27.659818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:27.660544 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:27.660869 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:27.661494 (+ 625us) server_negotiation.cc:299] Negotiation successful
0923 21:01:27.661938 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"spinlock_wait_cycles":10112,"thread_start_us":472,"threads_started":1}
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:27.672432 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.659607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:58715)
0923 21:01:27.660385 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:27.660406 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:27.660554 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:27.661860 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:27.661865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:27.662482 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:27.662772 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.662785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.664268 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.664274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.665807 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.665824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.667023 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.667029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.667228 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.667955 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:27.667981 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:27.672075 (+ 4094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":531,"threads_started":1}
W20260923 21:01:27.672624 9389 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:27.659690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58715 (local address 127.0.97.62:40997)
0923 21:01:27.659946 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.659952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.660507 (+ 555us) server_negotiation.cc:407] Connection header received
0923 21:01:27.661489 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.661494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.661577 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.661690 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.662955 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.664059 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.666014 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.666661 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.666758 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.672383 (+ 5625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58715: BlockingRecv error: recv got EOF from 127.0.97.1:58715 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:27.674386 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.661767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:48537)
0923 21:01:27.662160 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:27.662185 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:27.662300 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:27.664618 (+ 2318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:27.664624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:27.665088 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:27.665283 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.665295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.667223 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.667230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.668855 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.668873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.669912 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.669918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.670071 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.670855 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:27.670881 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:27.674037 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275}
W20260923 21:01:27.674619 9655 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:27.662116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48537 (local address 127.0.97.62:40997)
0923 21:01:27.664209 (+ 2093us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.664214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.664232 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:01:27.664273 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.664277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.664352 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.664452 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.665455 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.667048 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.669037 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.669718 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.669881 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.674300 (+ 4419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48537: BlockingRecv error: recv got EOF from 127.0.97.3:48537 (error 108)
Metrics: {"server-negotiator.queue_time_us":1858,"thread_start_us":5656,"threads_started":1}
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:27 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:27.683944 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:27.675205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:41475)
0923 21:01:27.675381 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:27.675399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:27.675500 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:27.675833 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:27.675836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:27.676141 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:27.676289 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.676297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.677489 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.677492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.678685 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:27.678701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.679767 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.679773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:27.679926 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.680601 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:27.680620 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:27.683721 (+ 3101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 21:01:27.684291 9655 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:27.675343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41475 (local address 127.0.97.62:40997)
0923 21:01:27.675545 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:27.675549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:27.675567 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:01:27.675641 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:27.675644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:27.675705 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:27.675795 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:27.676434 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.677338 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.678853 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:27.679602 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:27.679717 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:27.683971 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41475: BlockingRecv error: recv got EOF from 127.0.97.3:41475 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 21:01:27.684345 9651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 21:01:28 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:28 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:28.166595 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:28.155858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:59323)
0923 21:01:28.156561 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:28.156577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:28.156759 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:28.157186 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:28.157192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:28.157699 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:28.157953 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:28.157962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.159500 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.159509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:28.160608 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:28.160622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.161497 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.161502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:28.161648 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:28.162536 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:28.162558 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:28.166220 (+ 3662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":443,"threads_started":1}
W20260923 21:01:28.166779 9655 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:28.156192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59323 (local address 127.0.97.62:40997)
0923 21:01:28.156464 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:28.156470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:28.156672 (+ 202us) server_negotiation.cc:407] Connection header received
0923 21:01:28.156903 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:28.156915 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:28.156977 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:28.157068 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:28.158118 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.159214 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.160779 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.161289 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.161412 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:28.166566 (+ 5154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59323: BlockingRecv error: recv got EOF from 127.0.97.2:59323 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 23 21:01:28 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:28 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:28.683540 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:28.673922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:60309)
0923 21:01:28.674501 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:28.674523 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:28.674646 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:28.675329 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:28.675334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:28.675815 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:28.676027 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:28.676041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.677133 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.677138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:28.678106 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:28.678119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.678957 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.678961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:28.679090 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:28.679757 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:28.679777 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:28.683154 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":295,"threads_started":1}
W20260923 21:01:28.683728 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:28.674228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60309 (local address 127.0.97.62:40997)
0923 21:01:28.674791 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:28.674795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:28.674810 (+ 15us) server_negotiation.cc:407] Connection header received
0923 21:01:28.674881 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:28.674885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:28.674974 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:28.675110 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:28.676200 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.676974 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.678254 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.678790 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.678892 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:28.683414 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60309: BlockingRecv error: recv got EOF from 127.0.97.1:60309 (error 108)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":315,"threads_started":1}
Sep 23 21:01:28 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:28 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:28.695792 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:28.684937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:42727)
0923 21:01:28.685480 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:28.685504 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:28.685628 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:28.685987 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:28.685991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:28.686568 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:28.686847 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:28.686995 (+ 148us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.688380 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.688384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:28.689552 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:28.689568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.690655 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.690660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:28.690798 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:28.691711 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:28.691743 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:28.695518 (+ 3775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":293,"threads_started":1}
W20260923 21:01:28.696365 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:28.685032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42727 (local address 127.0.97.62:40997)
0923 21:01:28.685198 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:28.685202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:28.685582 (+ 380us) server_negotiation.cc:407] Connection header received
0923 21:01:28.685747 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:28.685751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:28.685819 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:28.685906 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:28.687158 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.688210 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.689744 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:28.690272 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:28.690579 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:28.696094 (+ 5515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42727: BlockingRecv error: recv got EOF from 127.0.97.3:42727 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 23 21:01:29 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:29 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:29.176404 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:29.167578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:55483)
0923 21:01:29.168217 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:29.168234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:29.168403 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:29.168777 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:29.168782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:29.169233 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:29.169381 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:29.169390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.170480 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.170487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:29.171444 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:29.171457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.172184 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.172189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:29.172361 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:29.173061 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:29.173079 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:29.176150 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":428,"threads_started":1}
W20260923 21:01:29.176826 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:29.167661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55483 (local address 127.0.97.62:40997)
0923 21:01:29.167895 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:29.167902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:29.168335 (+ 433us) server_negotiation.cc:407] Connection header received
0923 21:01:29.168570 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:29.168575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:29.168642 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:29.168748 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:29.169545 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.170283 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.171598 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.172011 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.172143 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:29.176496 (+ 4353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55483: BlockingRecv error: recv got EOF from 127.0.97.2:55483 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 21:01:29 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:29 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:29.694208 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:29.684612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:41325)
0923 21:01:29.685473 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:29.685490 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:29.685601 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:29.686003 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:29.686007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:29.686393 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:29.686561 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:29.686569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.687664 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.687668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:29.688622 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:29.688635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.689509 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.689515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:29.689721 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:29.690473 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:29.690492 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:29.693834 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":609,"threads_started":1}
W20260923 21:01:29.694420 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:29.684752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41325 (local address 127.0.97.62:40997)
0923 21:01:29.685304 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:29.685310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:29.685578 (+ 268us) server_negotiation.cc:407] Connection header received
0923 21:01:29.685751 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:29.685756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:29.685847 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:29.685948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:29.686729 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.687509 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.688783 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.689315 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.689429 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:29.694110 (+ 4681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41325: BlockingRecv error: recv got EOF from 127.0.97.1:41325 (error 108)
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":262,"threads_started":1}
Sep 23 21:01:29 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:29 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:29.706259 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:29.696872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:40731)
0923 21:01:29.697677 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:29.697694 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:29.697803 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:29.698277 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:29.698280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:29.698772 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:29.698936 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:29.698944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.699980 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.699984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:29.701049 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:29.701062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.701765 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.701770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:29.701911 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:29.702643 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:29.702662 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:29.706006 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":551,"threads_started":1}
W20260923 21:01:29.706399 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:29.696921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40731 (local address 127.0.97.62:40997)
0923 21:01:29.697113 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:29.697118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:29.697771 (+ 653us) server_negotiation.cc:407] Connection header received
0923 21:01:29.697997 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:29.698000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:29.698064 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:29.698153 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:29.699093 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.699831 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.701201 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:29.701593 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:29.701702 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:29.706154 (+ 4452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40731: BlockingRecv error: recv got EOF from 127.0.97.3:40731 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 21:01:30 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:30 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:30.187500 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:30.177633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:59037)
0923 21:01:30.178196 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:30.178214 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:30.178393 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:30.178880 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:30.178885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:30.179262 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:30.179464 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:30.179475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.180720 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.180724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:30.181785 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:30.181800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.182597 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.182602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:30.182895 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:30.183627 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:30.183645 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:30.187241 (+ 3596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":357,"threads_started":1}
W20260923 21:01:30.188020 9664 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:30.177852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59037 (local address 127.0.97.62:40997)
0923 21:01:30.178109 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:30.178117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:30.178428 (+ 311us) server_negotiation.cc:407] Connection header received
0923 21:01:30.178500 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:30.178504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:30.178586 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:30.178681 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:30.179663 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.180556 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.181976 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.182421 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.182558 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:30.187699 (+ 5141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59037: BlockingRecv error: recv got EOF from 127.0.97.2:59037 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 21:01:30 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:30 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:30.705327 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:30.695348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:56705)
0923 21:01:30.696008 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:30.696030 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:30.696180 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:30.696694 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:30.696698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:30.697221 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:30.697458 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:30.697472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.699008 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.699014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:30.700261 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:30.700276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.701052 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.701059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:30.701234 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:30.701940 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:30.701960 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:30.705082 (+ 3122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":299,"threads_started":1}
W20260923 21:01:30.705785 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:30.695548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56705 (local address 127.0.97.62:40997)
0923 21:01:30.696368 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:30.696372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:30.696389 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:01:30.696443 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:30.696446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:30.696514 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:30.696610 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:30.697693 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.698780 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.700404 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.700852 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.701024 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:30.705489 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56705: BlockingRecv error: recv got EOF from 127.0.97.1:56705 (error 108)
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":592,"threads_started":1}
Sep 23 21:01:30 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:30 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:30.716921 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:30.707605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:34127)
0923 21:01:30.708252 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:30.708270 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:30.708432 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:30.708770 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:30.708774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:30.709137 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:30.709298 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:30.709307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.710422 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.710426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:30.711568 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:30.711582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.712368 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.712373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:30.712549 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:30.713209 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:30.713227 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:30.716435 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":395,"threads_started":1}
W20260923 21:01:30.717003 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:30.707777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34127 (local address 127.0.97.62:40997)
0923 21:01:30.708075 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:30.708085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:30.708358 (+ 273us) server_negotiation.cc:407] Connection header received
0923 21:01:30.708574 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:30.708578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:30.708638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:30.708740 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:30.709449 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.710207 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.711728 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:30.712210 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:30.712343 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:30.716732 (+ 4389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34127: BlockingRecv error: recv got EOF from 127.0.97.3:34127 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 23 21:01:31 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:31 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:31.203346 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:31.188542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:56263)
0923 21:01:31.189070 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:31.189088 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:31.189271 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:31.189670 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:31.189674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:31.190080 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:31.190245 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:31.190253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.191428 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.191455 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:31.192489 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:31.192501 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.193379 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.193383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:31.193522 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:31.199066 (+ 5544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:31.199091 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:31.202929 (+ 3838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":310,"threads_started":1}
W20260923 21:01:31.203434 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:31.188624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56263 (local address 127.0.97.62:40997)
0923 21:01:31.188837 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:31.188845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:31.189188 (+ 343us) server_negotiation.cc:407] Connection header received
0923 21:01:31.189412 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:31.189431 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:31.189518 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:31.189628 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:31.190447 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.191279 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.192685 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.193232 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.193342 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:31.203147 (+ 9805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:56263: BlockingRecv error: recv got EOF from 127.0.97.2:56263 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 21:01:31 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:31 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:31.715909 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:31.706661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:55267)
0923 21:01:31.707193 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:31.707210 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:31.707316 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:31.707743 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:31.707748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:31.708322 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:31.708545 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:31.708556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.709703 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.709707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:31.710794 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:31.710807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.711534 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.711538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:31.711654 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:31.712289 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:31.712308 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:31.715563 (+ 3255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":280,"threads_started":1}
W20260923 21:01:31.716391 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:31.706787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55267 (local address 127.0.97.62:40997)
0923 21:01:31.707349 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:31.707355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:31.707374 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:01:31.707447 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:31.707451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:31.707537 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:31.707653 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:31.708692 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.709568 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.710966 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.711396 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.711504 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:31.716065 (+ 4561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55267: BlockingRecv error: recv got EOF from 127.0.97.1:55267 (error 108)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":425,"threads_started":1}
Sep 23 21:01:31 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:31 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:31.727903 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:31.717917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:58827)
0923 21:01:31.718483 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:31.718499 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:31.718615 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:31.719077 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:31.719080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:31.719421 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:31.719591 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:31.719600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.721046 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.721052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:31.722713 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:31.722730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.723654 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.723660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:31.723852 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:31.724536 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:31.724553 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:31.727629 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":309,"threads_started":1}
W20260923 21:01:31.728327 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:31.718118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58827 (local address 127.0.97.62:40997)
0923 21:01:31.718286 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:31.718290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:31.718573 (+ 283us) server_negotiation.cc:407] Connection header received
0923 21:01:31.718715 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:31.718718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:31.718779 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:31.718979 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:31.719747 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.720732 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.722877 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:31.723492 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:31.723641 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:31.728067 (+ 4426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58827: BlockingRecv error: recv got EOF from 127.0.97.3:58827 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 21:01:32 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:32 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 21:01:32.217005 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:32.204989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43485 (local address 127.0.97.62:40997)
0923 21:01:32.205254 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:32.205260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:32.205473 (+ 213us) server_negotiation.cc:407] Connection header received
0923 21:01:32.205686 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:32.205700 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:32.205895 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:32.206010 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:32.207186 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.208210 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.209532 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.209996 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.210144 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:32.216732 (+ 6588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43485: BlockingRecv error: recv got EOF from 127.0.97.2:43485 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 21:01:32.216998 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:32.204670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:43485)
0923 21:01:32.205359 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:32.205375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:32.205548 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:32.206123 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:32.206130 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:32.206738 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:32.207025 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:32.207036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.208400 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.208407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:32.209368 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:32.209381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.210162 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.210168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:32.210417 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:32.211374 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:32.211402 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:32.216460 (+ 5058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":343,"threads_started":1}
Sep 23 21:01:32 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:32 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:32.727476 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:32.717127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:42991)
0923 21:01:32.718887 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:32.718909 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:32.719063 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:32.719416 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:32.719420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:32.719860 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:32.720013 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:32.720022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.721122 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.721131 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:32.722596 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:32.722611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.723368 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.723372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:32.723493 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:32.724069 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:32.724087 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:32.727096 (+ 3009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1554,"thread_start_us":1431,"threads_started":1}
W20260923 21:01:32.727545 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:32.717188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42991 (local address 127.0.97.62:40997)
0923 21:01:32.717374 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:32.717403 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:32.719038 (+ 1635us) server_negotiation.cc:407] Connection header received
0923 21:01:32.719178 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:32.719182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:32.719276 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:32.719370 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:32.720150 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.720873 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.722739 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.723205 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.723305 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:32.727318 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42991: BlockingRecv error: recv got EOF from 127.0.97.1:42991 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 21:01:32 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:32 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:32.738139 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:32.728975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:55967)
0923 21:01:32.729508 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:32.729524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:32.729635 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:32.730017 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:32.730021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:32.730534 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:32.730783 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:32.730795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.731915 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.731920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:32.732955 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:32.732969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.733947 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.733953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:32.734170 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:32.734937 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:32.734961 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:32.737896 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":273,"threads_started":1}
W20260923 21:01:32.738336 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:32.729082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55967 (local address 127.0.97.62:40997)
0923 21:01:32.729272 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:32.729276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:32.729613 (+ 337us) server_negotiation.cc:407] Connection header received
0923 21:01:32.729739 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:32.729743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:32.729805 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:32.729897 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:32.730941 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.731741 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.733164 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:32.733658 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:32.733810 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:32.738105 (+ 4295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55967: BlockingRecv error: recv got EOF from 127.0.97.3:55967 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 21:01:33 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:33 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:33.227981 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:33.218401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:39479)
0923 21:01:33.219163 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:33.219183 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:33.219312 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:33.219661 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:33.219666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:33.220176 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:33.220355 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:33.220365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.221498 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.221502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:33.222717 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:33.222732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.223550 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.223556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:33.223704 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:33.224415 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:33.224435 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:33.227682 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":505,"threads_started":1}
W20260923 21:01:33.228183 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:33.218421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39479 (local address 127.0.97.62:40997)
0923 21:01:33.218601 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:33.218607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:33.219321 (+ 714us) server_negotiation.cc:407] Connection header received
0923 21:01:33.219417 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:33.219420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:33.219481 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:33.219569 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:33.220521 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.221320 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.222870 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.223320 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.223433 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:33.227921 (+ 4488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39479: BlockingRecv error: recv got EOF from 127.0.97.2:39479 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 21:01:33 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:33 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:33.738497 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:33.728649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:60049)
0923 21:01:33.729243 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:33.729262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:33.729382 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:33.729968 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:33.729971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:33.730457 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:33.730646 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:33.730655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.731876 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.731881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:33.732921 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:33.732933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.733722 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.733727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:33.733854 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:33.734595 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:33.734619 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:33.738180 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":246,"threads_started":1}
W20260923 21:01:33.738790 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:33.728849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60049 (local address 127.0.97.62:40997)
0923 21:01:33.729513 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:33.729517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:33.729534 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:01:33.729584 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:33.729587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:33.729645 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:33.729738 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:33.730837 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.731598 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.733087 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.733540 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.733643 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:33.738513 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60049: BlockingRecv error: recv got EOF from 127.0.97.1:60049 (error 108)
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":487,"threads_started":1}
Sep 23 21:01:33 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:33 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:33.747526 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:33.739222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:47453)
0923 21:01:33.739772 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:33.739798 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:33.739926 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:33.740299 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:33.740302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:33.740660 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:33.740837 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:33.740845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.741835 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.741839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:33.742831 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:33.742843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.743521 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.743525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:33.743645 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:33.744310 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:33.744327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:33.747207 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":300,"threads_started":1}
W20260923 21:01:33.747653 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:33.739302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47453 (local address 127.0.97.62:40997)
0923 21:01:33.739493 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:33.739498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:33.739907 (+ 409us) server_negotiation.cc:407] Connection header received
0923 21:01:33.740067 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:33.740076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:33.740140 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:33.740226 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:33.740970 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.741689 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.742961 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:33.743369 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:33.743465 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:33.747418 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47453: BlockingRecv error: recv got EOF from 127.0.97.3:47453 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 21:01:34 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:34 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:34.238756 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:34.229193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:38151)
0923 21:01:34.229947 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:34.229969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:34.230126 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:34.230619 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:34.230623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:34.231092 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:34.231336 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:34.231357 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.232827 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.232831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:34.233787 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:34.233800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.234608 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.234613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:34.234816 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:34.235500 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:34.235519 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:34.238492 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":264,"threads_started":1}
W20260923 21:01:34.239046 9679 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:34.229189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38151 (local address 127.0.97.62:40997)
0923 21:01:34.229497 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:34.229505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:34.230085 (+ 580us) server_negotiation.cc:407] Connection header received
0923 21:01:34.230310 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:34.230317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:34.230452 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:34.230552 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:34.231516 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.232603 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.233936 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.234445 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.234559 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:34.238686 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38151: BlockingRecv error: recv got EOF from 127.0.97.2:38151 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 23 21:01:34 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:34 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:34.749385 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:34.739809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:46043)
0923 21:01:34.740339 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:34.740369 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:34.740498 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:34.741006 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:34.741009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:34.741409 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:34.741563 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:34.741571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.742816 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.742820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:34.743815 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:34.743832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.744680 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.744688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:34.744839 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:34.745536 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:34.745555 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:34.749050 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":286,"threads_started":1}
W20260923 21:01:34.749545 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:34.739958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46043 (local address 127.0.97.62:40997)
0923 21:01:34.740675 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:34.740679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:34.740696 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:01:34.740759 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:34.740763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:34.740841 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:34.740940 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:34.741719 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.742559 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.743985 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.744452 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.744562 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:34.749253 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46043: BlockingRecv error: recv got EOF from 127.0.97.1:46043 (error 108)
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":489,"threads_started":1}
Sep 23 21:01:34 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:34 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:34.758651 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:34.748566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:52637)
0923 21:01:34.750404 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:34.750429 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:34.750580 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:34.750907 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:34.750912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:34.751294 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:34.751454 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:34.751463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.752579 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.752584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:34.753557 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:34.753568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.754311 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.754317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:34.754512 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:34.755102 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:34.755120 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:34.758214 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1645,"thread_start_us":1458,"threads_started":1}
W20260923 21:01:34.758723 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:34.748784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52637 (local address 127.0.97.62:40997)
0923 21:01:34.749862 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:34.749868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:34.750509 (+ 641us) server_negotiation.cc:407] Connection header received
0923 21:01:34.750705 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:34.750715 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:34.750774 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:34.750883 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:34.751600 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.752423 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.753713 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:34.754147 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:34.754284 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:34.758477 (+ 4193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52637: BlockingRecv error: recv got EOF from 127.0.97.3:52637 (error 108)
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":3045,"threads_started":1}
Sep 23 21:01:35 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:35 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:35.250962 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:35.240343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:37597)
0923 21:01:35.240951 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:35.240969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:35.241107 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:35.241594 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:35.241598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:35.242097 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:35.242268 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:35.242278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.243781 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.243786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:35.244761 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:35.244775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.245971 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.245979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:35.246182 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:35.247031 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:35.247057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:35.250707 (+ 3650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":323,"threads_started":1}
W20260923 21:01:35.251320 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:35.240438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37597 (local address 127.0.97.62:40997)
0923 21:01:35.240662 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:35.240667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:35.241093 (+ 426us) server_negotiation.cc:407] Connection header received
0923 21:01:35.241309 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:35.241314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:35.241389 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:35.241508 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:35.242495 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.243504 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.244965 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.245589 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.245873 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:35.251020 (+ 5147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37597: BlockingRecv error: recv got EOF from 127.0.97.2:37597 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 23 21:01:35 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:35 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:35.760382 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:35.750668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:39165)
0923 21:01:35.751478 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:35.751494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:35.751657 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:35.752111 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:35.752116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:35.752622 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:35.752845 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:35.752856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.753958 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.753961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:35.754958 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:35.754970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.755783 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.755788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:35.755946 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:35.756838 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:35.756857 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:35.759981 (+ 3124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":504,"threads_started":1}
W20260923 21:01:35.760497 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:35.750899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39165 (local address 127.0.97.62:40997)
0923 21:01:35.751106 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:35.751114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:35.751581 (+ 467us) server_negotiation.cc:407] Connection header received
0923 21:01:35.751855 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:35.751861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:35.751938 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:35.752046 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:35.753025 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.753826 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.755113 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.755591 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.755719 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:35.760218 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39165: BlockingRecv error: recv got EOF from 127.0.97.1:39165 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 23 21:01:35 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:35 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:35.770493 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:35.759698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:44585)
0923 21:01:35.761445 (+ 1747us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:35.761463 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:35.761620 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:35.761990 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:35.761995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:35.762510 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:35.762662 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:35.762671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.764105 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.764111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:35.765534 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:35.765549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.766525 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.766531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:35.766658 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:35.767303 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:35.767321 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:35.770158 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1632,"thread_start_us":1527,"threads_started":1}
W20260923 21:01:35.770785 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:35.759854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44585 (local address 127.0.97.62:40997)
0923 21:01:35.760816 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:35.760822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:35.761562 (+ 740us) server_negotiation.cc:407] Connection header received
0923 21:01:35.761781 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:35.761785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:35.761855 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:35.761961 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:35.762809 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.763879 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.765808 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:35.766261 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:35.766466 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:35.770446 (+ 3980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44585: BlockingRecv error: recv got EOF from 127.0.97.3:44585 (error 108)
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":1844,"threads_started":1}
Sep 23 21:01:36 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:36 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:36.267172 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:36.252145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:34559)
0923 21:01:36.252740 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:36.252756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:36.252927 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:36.253282 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:36.253287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:36.253716 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:36.253886 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:36.253895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.254989 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.254996 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:36.255967 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:36.255978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.256759 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.256764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:36.256946 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:36.263077 (+ 6131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:36.263102 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:36.266922 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":285,"threads_started":1}
W20260923 21:01:36.267491 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:36.252210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34559 (local address 127.0.97.62:40997)
0923 21:01:36.252451 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:36.252459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:36.252849 (+ 390us) server_negotiation.cc:407] Connection header received
0923 21:01:36.253063 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:36.253069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:36.253143 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:36.253247 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:36.254041 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.254856 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.256116 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.256591 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.256736 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:36.267203 (+ 10467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34559: BlockingRecv error: recv got EOF from 127.0.97.2:34559 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 21:01:36 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:36 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:36.771310 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:36.761857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:42537)
0923 21:01:36.762509 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:36.762527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:36.762809 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:36.763210 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:36.763226 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:36.763676 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:36.763833 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:36.763843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.765010 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.765017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:36.766101 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:36.766115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.767002 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.767009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:36.767207 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:36.767820 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:36.767840 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:36.770990 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":370,"threads_started":1}
W20260923 21:01:36.771576 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:36.762041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42537 (local address 127.0.97.62:40997)
0923 21:01:36.762354 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:36.762361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:36.762622 (+ 261us) server_negotiation.cc:407] Connection header received
0923 21:01:36.762953 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:36.762960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:36.763057 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:36.763186 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:36.764001 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.764817 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.766266 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.766822 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.766983 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:36.771298 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42537: BlockingRecv error: recv got EOF from 127.0.97.1:42537 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 23 21:01:36 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:36 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:36.780992 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:36.771538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:47773)
0923 21:01:36.772347 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:36.772373 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:36.772542 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:36.772968 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:36.772975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:36.773429 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:36.773609 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:36.773699 (+ 90us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.774903 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.774909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:36.775879 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:36.775890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.776656 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.776662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:36.776837 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:36.777439 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:36.777457 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:36.780711 (+ 3254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":507,"threads_started":1}
W20260923 21:01:36.781050 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:36.771827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47773 (local address 127.0.97.62:40997)
0923 21:01:36.772014 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:36.772022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:36.772454 (+ 432us) server_negotiation.cc:407] Connection header received
0923 21:01:36.772686 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:36.772692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:36.772794 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:36.772896 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:36.773837 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.774772 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.776026 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:36.776477 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:36.776617 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:36.780847 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47773: BlockingRecv error: recv got EOF from 127.0.97.3:47773 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 23 21:01:37 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:37 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:37.279081 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:37.268405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:40419)
0923 21:01:37.269219 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:37.269246 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:37.269415 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:37.269838 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:37.269843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:37.270462 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:37.270660 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:37.270675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.271971 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.271977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:37.273028 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:37.273042 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.274163 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.274169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:37.274436 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:37.275069 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:37.275088 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:37.278797 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":506,"threads_started":1}
W20260923 21:01:37.279356 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:37.268541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40419 (local address 127.0.97.62:40997)
0923 21:01:37.268848 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:37.268856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:37.269354 (+ 498us) server_negotiation.cc:407] Connection header received
0923 21:01:37.269555 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:37.269560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:37.269647 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:37.269768 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:37.270836 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.271785 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.273245 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.273971 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.274151 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:37.279031 (+ 4880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40419: BlockingRecv error: recv got EOF from 127.0.97.2:40419 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 23 21:01:37 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:37 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:37.780936 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:37.772413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:49027)
0923 21:01:37.772931 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:37.772948 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:37.773069 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:37.773356 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:37.773359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:37.773758 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:37.773926 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:37.773934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.774962 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.774966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:37.775888 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:37.775899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.776762 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.776767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:37.776884 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:37.777496 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:37.777513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:37.780584 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":259,"threads_started":1}
W20260923 21:01:37.781260 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:37.772645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49027 (local address 127.0.97.62:40997)
0923 21:01:37.772955 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:37.772959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:37.773078 (+ 119us) server_negotiation.cc:407] Connection header received
0923 21:01:37.773158 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:37.773161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:37.773224 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:37.773315 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:37.774069 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.774831 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.776033 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.776601 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.776699 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:37.780958 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49027: BlockingRecv error: recv got EOF from 127.0.97.1:49027 (error 108)
Metrics: {"server-negotiator.queue_time_us":193}
Sep 23 21:01:37 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:37 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:37.791292 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:37.782044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:54847)
0923 21:01:37.782687 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:37.782710 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:37.782854 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:37.783271 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:37.783277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:37.783835 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:37.784020 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:37.784030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.785298 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.785302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:37.786424 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:37.786438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.787189 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.787194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:37.787325 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:37.788069 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:37.788102 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:37.790998 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":363,"threads_started":1}
W20260923 21:01:37.791684 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:37.782076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54847 (local address 127.0.97.62:40997)
0923 21:01:37.782302 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:37.782311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:37.782782 (+ 471us) server_negotiation.cc:407] Connection header received
0923 21:01:37.782999 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:37.783004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:37.783086 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:37.783191 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:37.784179 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.785152 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.786572 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:37.787008 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:37.787108 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:37.791272 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54847: BlockingRecv error: recv got EOF from 127.0.97.3:54847 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 21:01:38 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:38 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:38.290112 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:38.280226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:54471)
0923 21:01:38.280870 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:38.280898 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:38.281022 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:38.281405 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:38.281409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:38.281853 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:38.282034 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:38.282043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.283273 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.283277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:38.284243 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:38.284254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.285450 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.285459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:38.285748 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:38.286643 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:38.286669 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:38.289844 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":411,"threads_started":1}
W20260923 21:01:38.290458 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:38.280371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54471 (local address 127.0.97.62:40997)
0923 21:01:38.280572 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:38.280576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:38.280990 (+ 414us) server_negotiation.cc:407] Connection header received
0923 21:01:38.281154 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:38.281163 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:38.281256 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:38.281350 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:38.282280 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.283081 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.284455 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.285201 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.285350 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:38.290091 (+ 4741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54471: BlockingRecv error: recv got EOF from 127.0.97.2:54471 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 21:01:38 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:38 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:38.793992 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:38.783399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:57879)
0923 21:01:38.783963 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:38.783984 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:38.784146 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:38.784529 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:38.784533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:38.784970 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:38.785159 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:38.785168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.786317 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.786372 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:38.787698 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:38.787715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.788655 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.788661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:38.788860 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:38.789571 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:38.789599 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:38.793730 (+ 4131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":298,"threads_started":1}
W20260923 21:01:38.794520 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:38.783514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57879 (local address 127.0.97.62:40997)
0923 21:01:38.783866 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:38.783872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:38.784092 (+ 220us) server_negotiation.cc:407] Connection header received
0923 21:01:38.784291 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:38.784295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:38.784366 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:38.784455 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:38.785322 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.786116 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.787893 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.788449 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.788615 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:38.794087 (+ 5472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57879: BlockingRecv error: recv got EOF from 127.0.97.1:57879 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 21:01:38 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:38 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:38.803474 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:38.792335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:44477)
0923 21:01:38.794737 (+ 2402us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:38.794759 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:38.794900 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:38.795288 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:38.795293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:38.795752 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:38.795945 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:38.795954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.797106 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.797111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:38.798082 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:38.798094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.798944 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.798949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:38.799123 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:38.799801 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:38.799821 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:38.803114 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2272,"thread_start_us":2146,"threads_started":1}
W20260923 21:01:38.803721 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:38.792421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44477 (local address 127.0.97.62:40997)
0923 21:01:38.792967 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:38.792975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:38.794834 (+ 1859us) server_negotiation.cc:407] Connection header received
0923 21:01:38.795065 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:38.795071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:38.795143 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:38.795262 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:38.796108 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.796963 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.798256 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:38.798779 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:38.798921 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:38.803421 (+ 4500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44477: BlockingRecv error: recv got EOF from 127.0.97.3:44477 (error 108)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":271,"threads_started":1}
Sep 23 21:01:39 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:39 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:39.301254 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:39.291246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:46593)
0923 21:01:39.291821 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:39.291838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:39.291997 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:39.292423 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:39.292427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:39.292936 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:39.293099 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:39.293109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.294201 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.294205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:39.295721 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:39.295735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.296545 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.296549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:39.296767 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:39.297449 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:39.297467 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:39.300885 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":290,"threads_started":1}
W20260923 21:01:39.301381 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:39.291366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46593 (local address 127.0.97.62:40997)
0923 21:01:39.291663 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:39.291670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:39.291926 (+ 256us) server_negotiation.cc:407] Connection header received
0923 21:01:39.292153 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:39.292163 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:39.292231 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:39.292384 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:39.293287 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.294055 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.295903 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.296390 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.296508 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:39.301133 (+ 4625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46593: BlockingRecv error: recv got EOF from 127.0.97.2:46593 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 21:01:39 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:39 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:39.804324 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:39.795250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:37019)
0923 21:01:39.795915 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:39.795932 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:39.796117 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:39.796485 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:39.796490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:39.796907 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:39.797071 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:39.797079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.798163 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.798170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:39.799417 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:39.799429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.800191 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.800196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:39.800346 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:39.800978 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:39.801022 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:39.804000 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":420,"threads_started":1}
W20260923 21:01:39.804633 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:39.795331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37019 (local address 127.0.97.62:40997)
0923 21:01:39.795559 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:39.795565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:39.796036 (+ 471us) server_negotiation.cc:407] Connection header received
0923 21:01:39.796257 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:39.796267 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:39.796341 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:39.796452 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:39.797240 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.797985 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.799542 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.800021 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.800128 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:39.804290 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37019: BlockingRecv error: recv got EOF from 127.0.97.1:37019 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 21:01:39 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:39 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:39.814054 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:39.804543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:33729)
0923 21:01:39.805272 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:39.805288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:39.805408 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:39.805734 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:39.805738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:39.806125 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:39.806282 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:39.806291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.807359 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.807363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:39.808389 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:39.808402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.809165 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.809170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:39.809308 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:39.810099 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:39.810117 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:39.813816 (+ 3699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":499,"threads_started":1}
W20260923 21:01:39.814381 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:39.804720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33729 (local address 127.0.97.62:40997)
0923 21:01:39.805030 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:39.805035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:39.805371 (+ 336us) server_negotiation.cc:407] Connection header received
0923 21:01:39.805512 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:39.805516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:39.805577 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:39.805659 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:39.806448 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.807190 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.808540 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:39.808999 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:39.809102 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:39.814053 (+ 4951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33729: BlockingRecv error: recv got EOF from 127.0.97.3:33729 (error 108)
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":197}
Sep 23 21:01:40 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:40 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:40.312664 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:40.302471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:37191)
0923 21:01:40.303041 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:40.303064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:40.303181 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:40.303550 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:40.303557 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:40.304029 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:40.304249 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:40.304260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.305756 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.305761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:40.306918 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:40.306932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.308071 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.308078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:40.308292 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:40.309083 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:40.309110 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:40.312435 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
W20260923 21:01:40.312935 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:40.302687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37191 (local address 127.0.97.62:40997)
0923 21:01:40.302872 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:40.302877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:40.303144 (+ 267us) server_negotiation.cc:407] Connection header received
0923 21:01:40.303287 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:40.303290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:40.303350 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:40.303440 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:40.304428 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.305416 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.307110 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.307842 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.307973 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:40.312659 (+ 4686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37191: BlockingRecv error: recv got EOF from 127.0.97.2:37191 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 21:01:40 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:40 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:40.814069 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:40.805524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:33061)
0923 21:01:40.806029 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:40.806045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:40.806202 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:40.806582 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:40.806587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:40.806983 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:40.807165 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:40.807174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.808197 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.808203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:40.809118 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:40.809129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.809855 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.809861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:40.810036 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:40.810837 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:40.810856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:40.813720 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
W20260923 21:01:40.814174 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:40.805671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33061 (local address 127.0.97.62:40997)
0923 21:01:40.805893 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:40.805899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:40.806131 (+ 232us) server_negotiation.cc:407] Connection header received
0923 21:01:40.806355 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:40.806360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:40.806447 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:40.806551 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:40.807306 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.808044 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.809271 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.809699 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.809831 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:40.813947 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33061: BlockingRecv error: recv got EOF from 127.0.97.1:33061 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 21:01:40 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:40 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:40.824451 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:40.815225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:48369)
0923 21:01:40.815782 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:40.815809 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:40.815972 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:40.816405 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:40.816412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:40.816895 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:40.817143 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:40.817160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.818443 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.818448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:40.819473 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:40.819485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.820300 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.820304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:40.820420 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:40.821120 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:40.821138 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:40.824215 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":290,"threads_started":1}
W20260923 21:01:40.824740 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:40.815268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48369 (local address 127.0.97.62:40997)
0923 21:01:40.815478 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:40.815482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:40.815892 (+ 410us) server_negotiation.cc:407] Connection header received
0923 21:01:40.816170 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:40.816179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:40.816268 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:40.816350 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:40.817275 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.818223 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.819666 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:40.820120 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:40.820237 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:40.824434 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48369: BlockingRecv error: recv got EOF from 127.0.97.3:48369 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 23 21:01:41 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:41 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:41.328838 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:41.313722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:36575)
0923 21:01:41.314213 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:41.314230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:41.314420 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:41.314831 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:41.314834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:41.315174 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:41.315324 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:41.315332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.316520 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.316526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:41.317776 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:41.317788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.318502 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.318506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:41.318627 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:41.325296 (+ 6669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:41.325320 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:41.328557 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":257,"threads_started":1}
W20260923 21:01:41.329119 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:41.313822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36575 (local address 127.0.97.62:40997)
0923 21:01:41.314062 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:41.314070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:41.314377 (+ 307us) server_negotiation.cc:407] Connection header received
0923 21:01:41.314630 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:41.314638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:41.314700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:41.314783 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:41.315449 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.316300 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.317899 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.318311 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.318468 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:41.328800 (+ 10332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36575: BlockingRecv error: recv got EOF from 127.0.97.2:36575 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 21:01:41 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:41 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:41.827467 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:41.815878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.1:60865)
0923 21:01:41.816540 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:41.816557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:41.816678 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:41.817220 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:41.817228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:41.817699 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:41.817889 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:41.817897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.819379 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.819387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:41.820815 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:41.820830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.822028 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.822040 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:41.822220 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:41.823070 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:41.823104 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:41.827076 (+ 3972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":413,"threads_started":1}
W20260923 21:01:41.827826 9699 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:41.815930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60865 (local address 127.0.97.62:40997)
0923 21:01:41.816162 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:41.816167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:41.816675 (+ 508us) server_negotiation.cc:407] Connection header received
0923 21:01:41.816802 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:41.816806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:41.817039 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:41.817183 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:41.818061 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.819212 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.820997 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.821814 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.821992 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:41.827450 (+ 5458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60865: BlockingRecv error: recv got EOF from 127.0.97.1:60865 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 23 21:01:41 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:41 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:41.835896 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:41.825608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.3:60969)
0923 21:01:41.826172 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:41.826199 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:41.826393 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:41.827249 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:41.827254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:41.827695 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:41.827911 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:41.827926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.829014 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.829018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:41.830119 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:41.830131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.831121 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.831126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:41.831261 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:41.831938 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:41.831960 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:41.835564 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":247,"threads_started":1}
W20260923 21:01:41.836635 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:41.825676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60969 (local address 127.0.97.62:40997)
0923 21:01:41.826798 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:41.826803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:41.826823 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:01:41.826880 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:41.826885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:41.826954 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:41.827063 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:41.828053 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.828869 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.830293 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:41.830963 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:41.831065 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:41.836307 (+ 5242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60969: BlockingRecv error: recv got EOF from 127.0.97.3:60969 (error 108)
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":818,"threads_started":1}
Sep 23 21:01:42 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 21:01:42 dist-test-slave-8xn5 krb5kdc[8651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 21:01:42.339738 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:42.330122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40997 (local address 127.0.97.2:37347)
0923 21:01:42.330750 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:42.330769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:42.330902 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:42.331317 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:42.331321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:42.331827 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:42.332033 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:42.332043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.333146 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.333150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:42.334657 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:42.334675 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.335508 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.335514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:42.335648 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:42.336383 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:42.336403 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:42.339410 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:40997: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":404,"threads_started":1}
W20260923 21:01:42.339875 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:42.330397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37347 (local address 127.0.97.62:40997)
0923 21:01:42.330621 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:42.330628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:42.330858 (+ 230us) server_negotiation.cc:407] Connection header received
0923 21:01:42.331058 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:42.331062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:42.331149 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:42.331236 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:42.332189 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.332965 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.334813 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.335311 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.335411 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:42.339637 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37347: BlockingRecv error: recv got EOF from 127.0.97.2:37347 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:01:42.652498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9240
I20260923 21:01:42.666816 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:42.653796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.0.97.1:35567)
0923 21:01:42.654416 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:42.654425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:42.654454 (+ 29us) server_negotiation.cc:407] Connection header received
0923 21:01:42.654516 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:42.654522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:42.654753 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:42.654910 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:42.656443 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.657683 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.659198 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.659827 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.659948 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:42.662951 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:42.662962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:42.662979 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:42.663049 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:42.666190 (+ 3141us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 21:01:42.666577 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51504: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":297,"threads_started":1}
W20260923 21:01:42.666975 9712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51504: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 21:01:42.667166 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9240: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:35567: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 21:01:42.667407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9240
I20260923 21:01:42.684576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9391
I20260923 21:01:42.698616 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:42.685911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.0.97.2:42407)
0923 21:01:42.686655 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:42.686663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:42.686696 (+ 33us) server_negotiation.cc:407] Connection header received
0923 21:01:42.686762 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:42.686768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:42.686997 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:42.687163 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:42.688591 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.689765 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.691256 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.691901 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.692004 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:42.694971 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:42.694981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:42.694998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:42.695040 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:42.697984 (+ 2944us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 21:01:42.698279 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42400: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":305,"threads_started":1}
W20260923 21:01:42.698736 9713 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42400: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 21:01:42.698906 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9391: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42407: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 21:01:42.699079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9391
I20260923 21:01:42.716135 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9523
I20260923 21:01:42.729835 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:42.717414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.0.97.3:34301)
0923 21:01:42.718179 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:42.718186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:42.718213 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:01:42.718272 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:42.718278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:42.718572 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:42.718732 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:42.720248 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.721405 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.722915 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:42.723541 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:42.723658 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:42.726593 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:42.726606 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:42.726626 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:42.726671 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:42.729299 (+ 2628us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 21:01:42.729594 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52030: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":420,"threads_started":1}
W20260923 21:01:42.729961 9714 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52030: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 21:01:42.730104 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9523: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34301: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 21:01:42.730288 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9523
I20260923 21:01:42.747736 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9171
I20260923 21:01:42.749130 9233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:42.869452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9171
2026-09-23T21:01:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21651 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T21:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:42Z Disabled control of system clock
I20260923 21:01:42.950664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41223
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 21:01:43.305150 9719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:43.305553 9719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:43.305720 9719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:43.313416 9719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:43.313606 9719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:43.313678 9719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:43.313762 9719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:43.313828 9719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:43.323760 9719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:43.325369 9719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:43.327407 9719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:43.341063 9724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:43.341130 9725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:43.341104 9727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:43.342116 9719 server_base.cc:1034] running on GCE node
I20260923 21:01:43.343053 9719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:43.344761 9719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:43.345989 9719 hybrid_clock.cc:648] HybridClock initialized: now 1790197303345941 us; error 48 us; skew 500 ppm
I20260923 21:01:43.346443 9719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:43.349056 9719 webserver.cc:460] Webserver started at http://127.0.97.62:38523/ using document root <none> and password file <none>
I20260923 21:01:43.349812 9719 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:43.349911 9719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:43.350214 9719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:43.353688 9719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9"
format_stamp: "Formatted at 2026-09-23 21:01:43 on dist-test-slave-8xn5"
server_key: "47fac2a22e29c8263f3000a3b28f7a5f"
server_key_iv: "73734ab2080b57ba85cb95039dd188d2"
server_key_version: "encryptionkey@0"
I20260923 21:01:43.354682 9719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9"
format_stamp: "Formatted at 2026-09-23 21:01:43 on dist-test-slave-8xn5"
server_key: "47fac2a22e29c8263f3000a3b28f7a5f"
server_key_iv: "73734ab2080b57ba85cb95039dd188d2"
server_key_version: "encryptionkey@0"
I20260923 21:01:43.360551 9719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:01:43.364331 9733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:43.366000 9719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:01:43.366178 9719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9"
format_stamp: "Formatted at 2026-09-23 21:01:43 on dist-test-slave-8xn5"
server_key: "47fac2a22e29c8263f3000a3b28f7a5f"
server_key_iv: "73734ab2080b57ba85cb95039dd188d2"
server_key_version: "encryptionkey@0"
I20260923 21:01:43.366611 9719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:43.386369 9719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:43.387401 9719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:43.387712 9719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:43.409480 9719 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41223
I20260923 21:01:43.409456 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41223 every 8 connection(s)
I20260923 21:01:43.411099 9719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:01:43.414904 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9719
I20260923 21:01:43.415138 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:01:43.415486 9785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:43.415982 388 external_mini_cluster.cc:1428] Setting key 6dd0e8880403e20c151a2a8998a55075
I20260923 21:01:43.429929 9785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9: Bootstrap starting.
I20260923 21:01:43.432279 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:43.418433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.97.62:41223)
0923 21:01:43.419717 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:43.419729 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:43.419776 (+ 47us) server_negotiation.cc:407] Connection header received
0923 21:01:43.421956 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:43.421985 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:43.422677 (+ 692us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:43.423066 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:43.423722 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:43.425163 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:43.427164 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:43.428031 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:43.428198 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:43.429615 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:43.429624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:43.429642 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:43.430391 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:43.430491 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:43.430515 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:43.430620 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:43.430858 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 21:01:43.431252 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":510,"threads_started":1}
I20260923 21:01:43.433529 9785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:43.434713 9785 log.cc:824] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:43.437918 9785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9: No bootstrap required, opened a new log
I20260923 21:01:43.445408 9785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41223 } }
I20260923 21:01:43.445853 9785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:43.445966 9785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e0d8f0a35a448c9a6c13ff01af0ae9, State: Initialized, Role: FOLLOWER
I20260923 21:01:43.446620 9785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41223 } }
I20260923 21:01:43.446863 9785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:01:43.447006 9785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:01:43.447185 9785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:43.449366 9785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41223 } }
I20260923 21:01:43.449898 9785 leader_election.cc:302] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e0d8f0a35a448c9a6c13ff01af0ae9; no voters:
I20260923 21:01:43.450634 9785 leader_election.cc:288] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:01:43.450804 9790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:43.451797 9790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [term 1 LEADER]: Becoming Leader. State: Replica: 20e0d8f0a35a448c9a6c13ff01af0ae9, State: Running, Role: LEADER
I20260923 21:01:43.452368 9790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41223 } }
I20260923 21:01:43.452873 9785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:43.455549 9792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e0d8f0a35a448c9a6c13ff01af0ae9. Latest consensus state: current_term: 1 leader_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41223 } } }
I20260923 21:01:43.455705 9791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e0d8f0a35a448c9a6c13ff01af0ae9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41223 } } }
I20260923 21:01:43.456014 9792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:43.456014 9791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:43.458305 9798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:43.464560 9798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:43.476014 9798 catalog_manager.cc:1347] Generated new cluster ID: 374b389d661243699c98ee2af17112c7
I20260923 21:01:43.476166 9798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:43.505249 9798 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:01:43.507439 9798 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:43.516780 9798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e0d8f0a35a448c9a6c13ff01af0ae9: Generated new TSK 0
I20260923 21:01:43.517741 9798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:01:43.534991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41223
--builtin_ntp_servers=127.0.97.20:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 21:01:43.913054 9809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:43.913424 9809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:43.913583 9809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:43.922241 9809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:43.922470 9809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:43.922669 9809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:43.933730 9809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:43.935307 9809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:43.937402 9809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:43.953502 9815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:43.953837 9814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:43.954365 9817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:43.954418 9809 server_base.cc:1034] running on GCE node
I20260923 21:01:43.961021 9809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:43.962303 9809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:43.963667 9809 hybrid_clock.cc:648] HybridClock initialized: now 1790197303963615 us; error 52 us; skew 500 ppm
I20260923 21:01:43.964068 9809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:43.967005 9809 webserver.cc:460] Webserver started at http://127.0.97.1:46257/ using document root <none> and password file <none>
I20260923 21:01:43.967787 9809 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:43.967924 9809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:43.968268 9809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:43.971508 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "1c37e8280bef4b96a1b413850a5b5ec7"
format_stamp: "Formatted at 2026-09-23 21:01:43 on dist-test-slave-8xn5"
server_key: "c30e0b10e762eb3a022e8043fe1e8bfe"
server_key_iv: "3b25358c245a5dc7d4fd6e09c7ff27ec"
server_key_version: "encryptionkey@0"
I20260923 21:01:43.972369 9809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1c37e8280bef4b96a1b413850a5b5ec7"
format_stamp: "Formatted at 2026-09-23 21:01:43 on dist-test-slave-8xn5"
server_key: "c30e0b10e762eb3a022e8043fe1e8bfe"
server_key_iv: "3b25358c245a5dc7d4fd6e09c7ff27ec"
server_key_version: "encryptionkey@0"
I20260923 21:01:43.978035 9809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 21:01:43.982029 9823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:43.983775 9809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:01:43.984009 9809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "1c37e8280bef4b96a1b413850a5b5ec7"
format_stamp: "Formatted at 2026-09-23 21:01:43 on dist-test-slave-8xn5"
server_key: "c30e0b10e762eb3a022e8043fe1e8bfe"
server_key_iv: "3b25358c245a5dc7d4fd6e09c7ff27ec"
server_key_version: "encryptionkey@0"
I20260923 21:01:43.984258 9809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:43.998864 9809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:43.999802 9809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:44.000125 9809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:44.001217 9809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:44.002909 9809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:44.003048 9809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.003188 9809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:44.003276 9809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.038317 9809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36651
I20260923 21:01:44.038444 9935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36651 every 8 connection(s)
I20260923 21:01:44.039753 9809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:44.041841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9809
I20260923 21:01:44.042057 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:01:44.042928 388 external_mini_cluster.cc:1428] Setting key e924213acd48c1102804aa69d434a1d4
I20260923 21:01:44.045961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41223
--builtin_ntp_servers=127.0.97.20:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 21:01:44.053578 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:44.042811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40953 (local address 127.0.97.62:41223)
0923 21:01:44.043274 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:44.043279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:44.044772 (+ 1493us) server_negotiation.cc:407] Connection header received
0923 21:01:44.046830 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:44.046835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:44.046921 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:44.047059 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:44.048170 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.048883 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.050557 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.050961 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.051102 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:44.052564 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:44.052569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:44.052572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:44.052620 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:44.052700 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:44.052714 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:44.052797 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:44.053107 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 21:01:44.053204 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":237,"threads_started":1}
I20260923 21:01:44.055075 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:44.043581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41223 (local address 127.0.97.1:40953)
0923 21:01:44.044620 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:44.044658 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:44.046545 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:44.047314 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:44.047323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:44.047419 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:44.047993 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:44.048008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.049060 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.049064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:44.050322 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:44.050401 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.051197 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.051204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:44.051447 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:44.052204 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:01:44.052207 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:44.052248 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:01:44.052376 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:44.052821 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:44.052834 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:44.052846 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 21:01:44.053055 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 21:01:44.053509 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":337,"threads_started":1}
I20260923 21:01:44.056779 9936 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41223
I20260923 21:01:44.057106 9936 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:44.058127 9936 heartbeater.cc:511] Master 127.0.97.62:41223 requested a full tablet report, sending...
I20260923 21:01:44.060629 9750 ts_manager.cc:194] Registered new tserver with Master: 1c37e8280bef4b96a1b413850a5b5ec7 (127.0.97.1:36651)
I20260923 21:01:44.062259 9750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40953
W20260923 21:01:44.395534 9940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:44.395926 9940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:44.396095 9940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:44.405085 9940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:44.405301 9940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:44.405445 9940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:44.415083 9940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:44.416428 9940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:44.418294 9940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:44.432796 9948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:44.432734 9946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:44.432802 9945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:44.436381 9940 server_base.cc:1034] running on GCE node
I20260923 21:01:44.437150 9940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:44.438246 9940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:44.439564 9940 hybrid_clock.cc:648] HybridClock initialized: now 1790197304439482 us; error 49 us; skew 500 ppm
I20260923 21:01:44.440008 9940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:44.442889 9940 webserver.cc:460] Webserver started at http://127.0.97.2:33507/ using document root <none> and password file <none>
I20260923 21:01:44.443717 9940 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:44.443872 9940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:44.444242 9940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:44.447551 9940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "cef6da765a374d9bab0d93ad25624b10"
format_stamp: "Formatted at 2026-09-23 21:01:44 on dist-test-slave-8xn5"
server_key: "262df42cc20df96008a0b19ee293c31d"
server_key_iv: "aa4a395bd0094bead73f987531f8fdf5"
server_key_version: "encryptionkey@0"
I20260923 21:01:44.448525 9940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cef6da765a374d9bab0d93ad25624b10"
format_stamp: "Formatted at 2026-09-23 21:01:44 on dist-test-slave-8xn5"
server_key: "262df42cc20df96008a0b19ee293c31d"
server_key_iv: "aa4a395bd0094bead73f987531f8fdf5"
server_key_version: "encryptionkey@0"
I20260923 21:01:44.455008 9940 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260923 21:01:44.459321 9954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.461225 9940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:01:44.461459 9940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "cef6da765a374d9bab0d93ad25624b10"
format_stamp: "Formatted at 2026-09-23 21:01:44 on dist-test-slave-8xn5"
server_key: "262df42cc20df96008a0b19ee293c31d"
server_key_iv: "aa4a395bd0094bead73f987531f8fdf5"
server_key_version: "encryptionkey@0"
I20260923 21:01:44.461725 9940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:44.474947 9940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:44.475895 9940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:44.476204 9940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:44.477298 9940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:44.479058 9940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:44.479180 9940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.479315 9940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:44.479413 9940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.516094 9940 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35905
I20260923 21:01:44.516188 10066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35905 every 8 connection(s)
I20260923 21:01:44.517586 9940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:44.522151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 9940
I20260923 21:01:44.522373 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:01:44.522984 388 external_mini_cluster.cc:1428] Setting key 0c07de06e827d34a228a9bb4c8b9e937
I20260923 21:01:44.526754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41223
--builtin_ntp_servers=127.0.97.20:43553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 21:01:44.534355 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:44.520732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51045 (local address 127.0.97.62:41223)
0923 21:01:44.520972 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:44.520977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:44.522763 (+ 1786us) server_negotiation.cc:407] Connection header received
0923 21:01:44.525289 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:44.525296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:44.525380 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:44.525493 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:44.527213 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.528202 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.530814 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.531454 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.531588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:44.533359 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:44.533367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:44.533374 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:44.533400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:44.533440 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:44.533456 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:44.533542 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:44.533821 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 21:01:44.533937 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 21:01:44.535728 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:44.521251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41223 (local address 127.0.97.2:51045)
0923 21:01:44.522553 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:44.522617 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:44.524954 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:44.525850 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:44.525866 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:44.526012 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:44.526977 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:44.527009 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.528389 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.528398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:44.530661 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:44.530679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.531705 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.531715 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:44.532082 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:44.532942 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:01:44.532947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:44.533014 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:01:44.533133 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:44.533561 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:44.533574 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:44.533586 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 21:01:44.533785 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 21:01:44.534290 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":350,"threads_started":1}
I20260923 21:01:44.537372 10067 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41223
I20260923 21:01:44.537717 10067 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:44.538556 10067 heartbeater.cc:511] Master 127.0.97.62:41223 requested a full tablet report, sending...
I20260923 21:01:44.540154 9750 ts_manager.cc:194] Registered new tserver with Master: cef6da765a374d9bab0d93ad25624b10 (127.0.97.2:35905)
I20260923 21:01:44.540992 9750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:51045
W20260923 21:01:44.870682 10070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:44.871096 10070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:44.871261 10070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:44.878685 10070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:44.878840 10070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:44.879036 10070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:44.888530 10070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41223
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:44.889977 10070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:44.891949 10070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:44.906584 10075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:44.906608 10078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:44.906584 10076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:44.907557 10070 server_base.cc:1034] running on GCE node
I20260923 21:01:44.908239 10070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:44.909273 10070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:44.910540 10070 hybrid_clock.cc:648] HybridClock initialized: now 1790197304910453 us; error 49 us; skew 500 ppm
I20260923 21:01:44.910878 10070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:44.913265 10070 webserver.cc:460] Webserver started at http://127.0.97.3:43471/ using document root <none> and password file <none>
I20260923 21:01:44.914007 10070 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:44.914144 10070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:44.914552 10070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:44.917594 10070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "b2232538300f465a96afa04158bf94ef"
format_stamp: "Formatted at 2026-09-23 21:01:44 on dist-test-slave-8xn5"
server_key: "e03d5c0023ff12dac2018474902a6e30"
server_key_iv: "ac0d9e932d8503a534a5d54b3546c1d2"
server_key_version: "encryptionkey@0"
I20260923 21:01:44.918545 10070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b2232538300f465a96afa04158bf94ef"
format_stamp: "Formatted at 2026-09-23 21:01:44 on dist-test-slave-8xn5"
server_key: "e03d5c0023ff12dac2018474902a6e30"
server_key_iv: "ac0d9e932d8503a534a5d54b3546c1d2"
server_key_version: "encryptionkey@0"
I20260923 21:01:44.923903 10070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:01:44.927788 10084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.929253 10070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:01:44.929450 10070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "b2232538300f465a96afa04158bf94ef"
format_stamp: "Formatted at 2026-09-23 21:01:44 on dist-test-slave-8xn5"
server_key: "e03d5c0023ff12dac2018474902a6e30"
server_key_iv: "ac0d9e932d8503a534a5d54b3546c1d2"
server_key_version: "encryptionkey@0"
I20260923 21:01:44.929687 10070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:44.944751 10070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:44.945669 10070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:44.945958 10070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:44.947023 10070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:44.948607 10070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:44.948752 10070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.948896 10070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:44.948982 10070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:44.984504 10070 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32963
I20260923 21:01:44.984557 10196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32963 every 8 connection(s)
I20260923 21:01:44.985925 10070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:44.993472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10070
I20260923 21:01:44.993661 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:01:44.994108 388 external_mini_cluster.cc:1428] Setting key ca17762a09d538f0e82bae5eba00441a
I20260923 21:01:45.000587 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:44.988936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55817 (local address 127.0.97.62:41223)
0923 21:01:44.989245 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:44.989250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:44.990877 (+ 1627us) server_negotiation.cc:407] Connection header received
0923 21:01:44.992852 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:44.992858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:44.992961 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:44.993086 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:44.994724 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.995837 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.997595 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.998047 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.998193 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:44.999623 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:44.999632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:44.999643 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:44.999661 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:44.999691 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:44.999703 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:44.999803 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:45.000070 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 21:01:45.000197 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260923 21:01:45.001690 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:44.989602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41223 (local address 127.0.97.3:55817)
0923 21:01:44.990695 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:44.990741 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:44.992542 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:44.993376 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:44.993387 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:44.993496 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:44.994457 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:44.994489 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.996030 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.996036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:44.997435 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:44.997448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:44.998297 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:44.998306 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:44.998599 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:44.999264 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:01:44.999267 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:44.999323 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:01:44.999410 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:44.999815 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:44.999828 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:44.999837 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 21:01:45.000047 (+ 210us) client_negotiation.cc:240] Negotiation successful
0923 21:01:45.000506 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":317,"threads_started":1}
I20260923 21:01:45.003214 10197 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41223
I20260923 21:01:45.003515 10197 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:45.004319 10197 heartbeater.cc:511] Master 127.0.97.62:41223 requested a full tablet report, sending...
I20260923 21:01:45.005791 9750 ts_manager.cc:194] Registered new tserver with Master: b2232538300f465a96afa04158bf94ef (127.0.97.3:32963)
I20260923 21:01:45.006701 9750 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:55817
I20260923 21:01:45.009464 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 21:01:45.016062 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:45.014293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.0.97.62:41223)
0923 21:01:45.014584 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:45.014589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:45.015042 (+ 453us) server_negotiation.cc:407] Connection header received
0923 21:01:45.015189 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:45.015193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:45.015261 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:45.015355 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:45.015871 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55998: BlockingRecv error: recv got EOF from 127.0.0.1:55998 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 21:01:45.016844 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9809
I20260923 21:01:45.025907 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:45.017965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.0.97.1:36651)
0923 21:01:45.018622 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:45.018629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:45.018656 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:01:45.018722 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:45.018729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:45.018952 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:45.019152 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:45.019668 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:45.020785 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:45.022133 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:45.022889 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:45.023064 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:45.024261 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:45.024270 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:45.024286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:45.024959 (+ 673us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:45.025022 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:45.025048 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:45.025189 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:45.025363 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 21:01:45.025533 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":289,"threads_started":1}
I20260923 21:01:45.027581 9931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:45.111079 9936 heartbeater.cc:503] Master 127.0.97.62:41223 was elected leader, sending a full tablet report...
I20260923 21:01:45.114607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9809
I20260923 21:01:45.131816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9940
I20260923 21:01:45.141288 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:45.132850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.0.97.2:35905)
0923 21:01:45.133409 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:45.133417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:45.133443 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:01:45.133510 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:45.133517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:45.133834 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:45.134032 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:45.134530 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:45.135664 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:45.137084 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:45.137922 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:45.138082 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:45.139466 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:45.139476 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:45.139491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:45.140241 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:45.140300 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:45.140332 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:45.140482 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:45.140749 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 21:01:45.140895 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260923 21:01:45.142688 10062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:45.226143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9940
I20260923 21:01:45.244534 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10070
I20260923 21:01:45.253552 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:45.245699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59994 (local address 127.0.97.3:32963)
0923 21:01:45.246463 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:45.246471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:45.246511 (+ 40us) server_negotiation.cc:407] Connection header received
0923 21:01:45.246586 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:45.246593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:45.246818 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:45.247033 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:45.247480 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:45.248509 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:45.249831 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:45.250534 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:45.250674 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:45.251922 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:45.251930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:45.251946 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:45.252559 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:45.252669 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:45.252696 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:45.252818 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:45.252987 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 21:01:45.253143 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":385,"threads_started":1}
I20260923 21:01:45.255046 10192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:45.337275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10070
I20260923 21:01:45.355262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 9719
I20260923 21:01:45.356900 9780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:45.466056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 9719
2026-09-23T21:01:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2590 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T21:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:45Z Disabled control of system clock
I20260923 21:01:45.540925 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44221
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 21:01:45.887020 10216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:45.887337 10216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:45.887547 10216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:45.895088 10216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:45.895203 10216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:45.895275 10216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:45.895368 10216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:45.895459 10216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:45.904896 10216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:45.906286 10216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:45.908156 10216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:45.920969 10221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:45.920922 10224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:45.920861 10222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:45.921728 10216 server_base.cc:1034] running on GCE node
I20260923 21:01:45.922567 10216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:45.924060 10216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:45.925290 10216 hybrid_clock.cc:648] HybridClock initialized: now 1790197305925241 us; error 58 us; skew 500 ppm
I20260923 21:01:45.925652 10216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:45.928082 10216 webserver.cc:460] Webserver started at http://127.0.97.62:40047/ using document root <none> and password file <none>
I20260923 21:01:45.928817 10216 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:45.928951 10216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:45.929288 10216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:45.934511 10216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "37cd84287af547ee81cf21e38e5ce60f"
format_stamp: "Formatted at 2026-09-23 21:01:45 on dist-test-slave-8xn5"
server_key: "09533773f6dcf717b6c0576d86bc4e46"
server_key_iv: "0e96619948f10fbeac841f76363cd549"
server_key_version: "encryptionkey@0"
I20260923 21:01:45.935410 10216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "37cd84287af547ee81cf21e38e5ce60f"
format_stamp: "Formatted at 2026-09-23 21:01:45 on dist-test-slave-8xn5"
server_key: "09533773f6dcf717b6c0576d86bc4e46"
server_key_iv: "0e96619948f10fbeac841f76363cd549"
server_key_version: "encryptionkey@0"
I20260923 21:01:45.941151 10216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:01:45.945120 10230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:45.946906 10216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:01:45.947125 10216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "37cd84287af547ee81cf21e38e5ce60f"
format_stamp: "Formatted at 2026-09-23 21:01:45 on dist-test-slave-8xn5"
server_key: "09533773f6dcf717b6c0576d86bc4e46"
server_key_iv: "0e96619948f10fbeac841f76363cd549"
server_key_version: "encryptionkey@0"
I20260923 21:01:45.947364 10216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:45.953320 10216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:45.954175 10216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:45.954541 10216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:45.973443 10216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44221
I20260923 21:01:45.973476 10281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44221 every 8 connection(s)
I20260923 21:01:45.974879 10216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:01:45.978765 10282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:45.985358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10216
I20260923 21:01:45.985616 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:01:45.986527 388 external_mini_cluster.cc:1428] Setting key 23791d59dcf6dd3d9cea7d47ac96646c
I20260923 21:01:45.990383 10282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f: Bootstrap starting.
I20260923 21:01:45.994233 10282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:45.995512 10282 log.cc:824] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:45.995913 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:45.988959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.0.97.62:44221)
0923 21:01:45.990053 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:45.990065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:45.990107 (+ 42us) server_negotiation.cc:407] Connection header received
0923 21:01:45.991949 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:45.991972 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:45.992459 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:45.992812 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:45.993982 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:45.993988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:45.994002 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:45.994016 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:45.994082 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:45.994119 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:45.994295 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:45.994631 (+ 336us) server_negotiation.cc:299] Negotiation successful
0923 21:01:45.994935 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":399,"threads_started":1}
I20260923 21:01:45.998518 10282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f: No bootstrap required, opened a new log
I20260923 21:01:46.005611 10282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cd84287af547ee81cf21e38e5ce60f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44221 } }
I20260923 21:01:46.006022 10282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:46.006135 10282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37cd84287af547ee81cf21e38e5ce60f, State: Initialized, Role: FOLLOWER
I20260923 21:01:46.006831 10282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cd84287af547ee81cf21e38e5ce60f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44221 } }
I20260923 21:01:46.007081 10282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:01:46.007228 10282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:01:46.007412 10282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:46.009765 10282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cd84287af547ee81cf21e38e5ce60f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44221 } }
I20260923 21:01:46.010301 10282 leader_election.cc:302] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37cd84287af547ee81cf21e38e5ce60f; no voters:
I20260923 21:01:46.011009 10282 leader_election.cc:288] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:01:46.011152 10287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:46.012615 10287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [term 1 LEADER]: Becoming Leader. State: Replica: 37cd84287af547ee81cf21e38e5ce60f, State: Running, Role: LEADER
I20260923 21:01:46.013238 10287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cd84287af547ee81cf21e38e5ce60f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44221 } }
I20260923 21:01:46.013638 10282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:46.016927 10288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37cd84287af547ee81cf21e38e5ce60f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cd84287af547ee81cf21e38e5ce60f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44221 } } }
I20260923 21:01:46.017309 10288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:46.016988 10289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37cd84287af547ee81cf21e38e5ce60f. Latest consensus state: current_term: 1 leader_uuid: "37cd84287af547ee81cf21e38e5ce60f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cd84287af547ee81cf21e38e5ce60f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44221 } } }
I20260923 21:01:46.017427 10289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:46.023200 10294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:46.030050 10294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:46.041640 10294 catalog_manager.cc:1347] Generated new cluster ID: d31b1d96412e4ce088b1875029741d1e
I20260923 21:01:46.041787 10294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:46.053920 10294 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:01:46.054466 10294 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:46.060603 10294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37cd84287af547ee81cf21e38e5ce60f: Generated new TSK 0
I20260923 21:01:46.061486 10294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:01:46.071099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44221
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 21:01:46.425429 10306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:46.425815 10306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:46.425992 10306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:46.435029 10306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:46.435168 10306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:46.435354 10306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:46.446450 10306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:46.447891 10306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:46.449855 10306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:46.466034 10311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:46.466120 10314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:46.466094 10312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:46.472386 10306 server_base.cc:1034] running on GCE node
I20260923 21:01:46.473155 10306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:46.474423 10306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:46.475656 10306 hybrid_clock.cc:648] HybridClock initialized: now 1790197306475610 us; error 51 us; skew 500 ppm
I20260923 21:01:46.476015 10306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:46.484107 10306 webserver.cc:460] Webserver started at http://127.0.97.1:40879/ using document root <none> and password file <none>
I20260923 21:01:46.484992 10306 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:46.485095 10306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:46.485430 10306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:46.488663 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "ccdef02be8d54c77974be5ebf43f220b"
format_stamp: "Formatted at 2026-09-23 21:01:46 on dist-test-slave-8xn5"
server_key: "6d540a798c5342145412a255fe4a4d97"
server_key_iv: "737a64e3dd13c51484e150ebb620f078"
server_key_version: "encryptionkey@0"
I20260923 21:01:46.489573 10306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ccdef02be8d54c77974be5ebf43f220b"
format_stamp: "Formatted at 2026-09-23 21:01:46 on dist-test-slave-8xn5"
server_key: "6d540a798c5342145412a255fe4a4d97"
server_key_iv: "737a64e3dd13c51484e150ebb620f078"
server_key_version: "encryptionkey@0"
I20260923 21:01:46.496055 10306 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 21:01:46.500274 10320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:46.501957 10306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:01:46.502192 10306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "ccdef02be8d54c77974be5ebf43f220b"
format_stamp: "Formatted at 2026-09-23 21:01:46 on dist-test-slave-8xn5"
server_key: "6d540a798c5342145412a255fe4a4d97"
server_key_iv: "737a64e3dd13c51484e150ebb620f078"
server_key_version: "encryptionkey@0"
I20260923 21:01:46.502568 10306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:46.509016 10306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:46.509946 10306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:46.510233 10306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:46.511222 10306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:46.512912 10306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:46.513036 10306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:46.513175 10306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:46.513262 10306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:46.549041 10306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38341
I20260923 21:01:46.549103 10432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38341 every 8 connection(s)
I20260923 21:01:46.550513 10306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:46.554920 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10306
I20260923 21:01:46.555157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:01:46.555789 388 external_mini_cluster.cc:1428] Setting key 477e2053a679683e7e38887fd46067bd
I20260923 21:01:46.559067 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44221
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 21:01:46.560129 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:46.553407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35707 (local address 127.0.97.62:44221)
0923 21:01:46.553849 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:46.553853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:46.555345 (+ 1492us) server_negotiation.cc:407] Connection header received
0923 21:01:46.557291 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:46.557296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:46.557361 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:46.557467 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:46.559294 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:46.559303 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:46.559311 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:46.559338 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:46.559407 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:46.559426 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:46.559601 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:46.559820 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 21:01:46.559878 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260923 21:01:46.561904 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:46.554081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44221 (local address 127.0.97.1:35707)
0923 21:01:46.555145 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:46.555184 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:46.557003 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:46.557766 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:46.557776 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:46.557921 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:46.558848 (+ 927us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:01:46.558852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:46.558911 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:01:46.559039 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:46.559577 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:46.559591 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:46.559603 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 21:01:46.559901 (+ 298us) client_negotiation.cc:240] Negotiation successful
0923 21:01:46.560219 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":365,"threads_started":1}
I20260923 21:01:46.563800 10433 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44221
I20260923 21:01:46.564155 10433 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:46.564810 10433 heartbeater.cc:511] Master 127.0.97.62:44221 requested a full tablet report, sending...
I20260923 21:01:46.567281 10247 ts_manager.cc:194] Registered new tserver with Master: ccdef02be8d54c77974be5ebf43f220b (127.0.97.1:38341)
W20260923 21:01:46.899466 10437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:46.899760 10437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:46.899852 10437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:46.907384 10437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:46.907503 10437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:46.907614 10437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:46.917032 10437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:46.918308 10437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:46.920085 10437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:46.934129 10442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:46.934046 10443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:46.934085 10445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:46.935400 10437 server_base.cc:1034] running on GCE node
I20260923 21:01:46.936092 10437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:46.937080 10437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:46.938395 10437 hybrid_clock.cc:648] HybridClock initialized: now 1790197306938273 us; error 70 us; skew 500 ppm
I20260923 21:01:46.938764 10437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:46.941445 10437 webserver.cc:460] Webserver started at http://127.0.97.2:35547/ using document root <none> and password file <none>
I20260923 21:01:46.942241 10437 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:46.942404 10437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:46.942749 10437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:46.945780 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "861b0d74dd464216b2310787c802c379"
format_stamp: "Formatted at 2026-09-23 21:01:46 on dist-test-slave-8xn5"
server_key: "9a7c83ebf07d320a2bcf4a6c29f4282f"
server_key_iv: "ff0b20c0f942691bafcc96e77f90c14e"
server_key_version: "encryptionkey@0"
I20260923 21:01:46.946833 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "861b0d74dd464216b2310787c802c379"
format_stamp: "Formatted at 2026-09-23 21:01:46 on dist-test-slave-8xn5"
server_key: "9a7c83ebf07d320a2bcf4a6c29f4282f"
server_key_iv: "ff0b20c0f942691bafcc96e77f90c14e"
server_key_version: "encryptionkey@0"
I20260923 21:01:46.952505 10437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260923 21:01:46.956378 10451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:46.957903 10437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:01:46.958122 10437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "861b0d74dd464216b2310787c802c379"
format_stamp: "Formatted at 2026-09-23 21:01:46 on dist-test-slave-8xn5"
server_key: "9a7c83ebf07d320a2bcf4a6c29f4282f"
server_key_iv: "ff0b20c0f942691bafcc96e77f90c14e"
server_key_version: "encryptionkey@0"
I20260923 21:01:46.958406 10437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:46.964978 10437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:46.965904 10437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:46.966205 10437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:46.967247 10437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:46.968731 10437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:46.968863 10437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:46.969007 10437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:46.969097 10437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:47.004426 10437 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42069
I20260923 21:01:47.004526 10563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42069 every 8 connection(s)
I20260923 21:01:47.005995 10437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:47.015273 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10437
I20260923 21:01:47.015424 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:47.009110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58849 (local address 127.0.97.62:44221)
0923 21:01:47.009348 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:47.009355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:47.011037 (+ 1682us) server_negotiation.cc:407] Connection header received
0923 21:01:47.012737 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:47.012742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:47.012796 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:47.012894 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:47.014377 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:47.014384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:47.014389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:47.014414 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:47.014464 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:47.014486 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:47.014619 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:47.014997 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 21:01:47.015086 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 21:01:47.015586 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:01:47.016077 388 external_mini_cluster.cc:1428] Setting key b056a9c1da57182001e5604603de0205
I20260923 21:01:47.016506 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:47.009757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44221 (local address 127.0.97.2:58849)
0923 21:01:47.010832 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:47.010878 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:47.012439 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:47.013135 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:47.013143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:47.013242 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:47.013927 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:01:47.013929 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:47.013992 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:01:47.014086 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:47.014650 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:47.014659 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:47.014666 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 21:01:47.015005 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 21:01:47.015238 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":342,"threads_started":1}
I20260923 21:01:47.018196 10564 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44221
I20260923 21:01:47.018573 10564 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:47.019294 10564 heartbeater.cc:511] Master 127.0.97.62:44221 requested a full tablet report, sending...
I20260923 21:01:47.019358 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44221
--builtin_ntp_servers=127.0.97.20:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 21:01:47.020962 10247 ts_manager.cc:194] Registered new tserver with Master: 861b0d74dd464216b2310787c802c379 (127.0.97.2:42069)
W20260923 21:01:47.361362 10567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:47.361756 10567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:47.361919 10567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:47.369632 10567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:47.369746 10567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:47.369928 10567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:47.379328 10567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:47.380733 10567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:47.382716 10567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:47.397445 10572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:47.397296 10573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:47.397246 10575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:47.398257 10567 server_base.cc:1034] running on GCE node
I20260923 21:01:47.399004 10567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:47.399935 10567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:47.401160 10567 hybrid_clock.cc:648] HybridClock initialized: now 1790197307401112 us; error 56 us; skew 500 ppm
I20260923 21:01:47.401559 10567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:01:47.404381 10567 webserver.cc:460] Webserver started at http://127.0.97.3:44443/ using document root <none> and password file <none>
I20260923 21:01:47.405133 10567 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:47.405248 10567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:47.405596 10567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:47.408777 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "ac111634ef614aa4b8d91fd998c2d049"
format_stamp: "Formatted at 2026-09-23 21:01:47 on dist-test-slave-8xn5"
server_key: "18d8527c5c7298f2455e86238d59275c"
server_key_iv: "7dea539f5e2df69b2d40dc52b12f176a"
server_key_version: "encryptionkey@0"
I20260923 21:01:47.409600 10567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ac111634ef614aa4b8d91fd998c2d049"
format_stamp: "Formatted at 2026-09-23 21:01:47 on dist-test-slave-8xn5"
server_key: "18d8527c5c7298f2455e86238d59275c"
server_key_iv: "7dea539f5e2df69b2d40dc52b12f176a"
server_key_version: "encryptionkey@0"
I20260923 21:01:47.415987 10567 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:01:47.420441 10581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:47.421972 10567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:01:47.422183 10567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "ac111634ef614aa4b8d91fd998c2d049"
format_stamp: "Formatted at 2026-09-23 21:01:47 on dist-test-slave-8xn5"
server_key: "18d8527c5c7298f2455e86238d59275c"
server_key_iv: "7dea539f5e2df69b2d40dc52b12f176a"
server_key_version: "encryptionkey@0"
I20260923 21:01:47.422595 10567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:47.429796 10567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:47.431186 10567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:47.431634 10567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:47.432936 10567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:47.434629 10567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:47.434727 10567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:47.434876 10567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:47.435007 10567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:47.473605 10567 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43055
I20260923 21:01:47.473649 10693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43055 every 8 connection(s)
I20260923 21:01:47.475265 10567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:47.484395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10567
I20260923 21:01:47.484607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:01:47.484701 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:47.477980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56353 (local address 127.0.97.62:44221)
0923 21:01:47.478177 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:47.478182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:47.479859 (+ 1677us) server_negotiation.cc:407] Connection header received
0923 21:01:47.481961 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:47.481967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:47.482057 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:47.482183 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:47.483807 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:47.483814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:47.483820 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:47.483856 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:47.483889 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:47.483902 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:47.484085 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:47.484295 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 21:01:47.484374 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:01:47.485162 388 external_mini_cluster.cc:1428] Setting key 32f278567658b2d86f74ac09a7730d76
I20260923 21:01:47.485848 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:47.478604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44221 (local address 127.0.97.3:56353)
0923 21:01:47.479656 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:47.479697 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:47.481600 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:47.482517 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:47.482526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:47.482641 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:47.483450 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:01:47.483453 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:47.483499 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:01:47.483597 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:47.484061 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:47.484071 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:47.484078 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 21:01:47.484336 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 21:01:47.484672 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":315,"threads_started":1}
I20260923 21:01:47.487661 10694 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44221
I20260923 21:01:47.487999 10694 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:47.488716 10694 heartbeater.cc:511] Master 127.0.97.62:44221 requested a full tablet report, sending...
I20260923 21:01:47.490273 10247 ts_manager.cc:194] Registered new tserver with Master: ac111634ef614aa4b8d91fd998c2d049 (127.0.97.3:43055)
I20260923 21:01:47.501492 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 21:01:47.508448 10435 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:01:47.506428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.0.97.62:44221)
0923 21:01:47.506662 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:47.506670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:47.507528 (+ 858us) server_negotiation.cc:407] Connection header received
0923 21:01:47.507751 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:47.507755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:47.507811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:47.507913 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:47.508244 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52500: BlockingRecv error: recv got EOF from 127.0.0.1:52500 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 21:01:47.509424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10306
I20260923 21:01:47.513417 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:47.510504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.0.97.1:38341)
0923 21:01:47.511015 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:47.511022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:47.511050 (+ 28us) server_negotiation.cc:407] Connection header received
0923 21:01:47.511119 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:47.511132 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:47.511441 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:47.511580 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:47.512550 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:47.512558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:47.512570 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:47.512594 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:47.512644 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:47.512659 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:47.512774 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:47.512988 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 21:01:47.513122 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":267,"threads_started":1}
I20260923 21:01:47.514871 10428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:47.596391 10433 heartbeater.cc:503] Master 127.0.97.62:44221 was elected leader, sending a full tablet report...
I20260923 21:01:47.597692 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10306
W20260923 21:01:47.614114 10232 connection.cc:535] server connection from 127.0.97.1:35707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 21:01:47.614475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10437
I20260923 21:01:47.618150 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:47.615438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.97.2:42069)
0923 21:01:47.615986 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:47.615994 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:47.616035 (+ 41us) server_negotiation.cc:407] Connection header received
0923 21:01:47.616104 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:47.616111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:47.616347 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:47.616543 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:47.617407 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:47.617414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:47.617427 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:47.617445 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:47.617491 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:47.617510 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:47.617624 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:47.617705 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 21:01:47.617848 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
I20260923 21:01:47.619664 10559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:47.703312 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10437
I20260923 21:01:47.720813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10567
I20260923 21:01:47.724887 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:47.721995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.0.97.3:43055)
0923 21:01:47.722702 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:47.722709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:47.722742 (+ 33us) server_negotiation.cc:407] Connection header received
0923 21:01:47.722803 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:47.722809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:47.723041 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:47.723172 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:47.724125 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:47.724131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:47.724143 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:01:47.724155 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:47.724194 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:01:47.724208 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:47.724381 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:47.724454 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 21:01:47.724535 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":393,"threads_started":1}
I20260923 21:01:47.726217 10689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:47.809906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10567
I20260923 21:01:47.825749 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10216
I20260923 21:01:47.827126 10277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:47.929206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10216
2026-09-23T21:01:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2464 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:01:48 dist-test-slave-8xn5 krb5kdc[10712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:01:48 dist-test-slave-8xn5 krb5kdc[10712](info): set up 2 sockets
Sep 23 21:01:48 dist-test-slave-8xn5 krb5kdc[10712](info): commencing operation
krb5kdc: starting...
W20260923 21:01:49.059448 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:01:49 dist-test-slave-8xn5 krb5kdc[10712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:01:49.315743 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35455
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:49.656499 10725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:49.656812 10725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:49.656972 10725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:49.664384 10725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:01:49.664516 10725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:49.664614 10725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:49.664702 10725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:01:49.664784 10725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:01:49.675582 10725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:49.676991 10725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:49.678380 10725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:49.690815 10731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:49.690843 10730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:49.691017 10733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:49.691847 10725 server_base.cc:1034] running on GCE node
I20260923 21:01:49.692664 10725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:49.694129 10725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:49.695441 10725 hybrid_clock.cc:648] HybridClock initialized: now 1790197309695367 us; error 85 us; skew 500 ppm
I20260923 21:01:49.695865 10725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:49 dist-test-slave-8xn5 krb5kdc[10712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:49.699124 10725 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:01:49.700688 10725 webserver.cc:460] Webserver started at http://127.0.97.62:35637/ using document root <none> and password file <none>
I20260923 21:01:49.701429 10725 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:49.701579 10725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:49.701891 10725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:49.705054 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "42dfe49cceaf444682d33ca3598a3b68"
format_stamp: "Formatted at 2026-09-23 21:01:49 on dist-test-slave-8xn5"
server_key: "5b7e55a08270f7918c4978356e2f2e76"
server_key_iv: "b5eb8aa1b09d11dfa12b0e3f439124de"
server_key_version: "encryptionkey@0"
I20260923 21:01:49.705850 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "42dfe49cceaf444682d33ca3598a3b68"
format_stamp: "Formatted at 2026-09-23 21:01:49 on dist-test-slave-8xn5"
server_key: "5b7e55a08270f7918c4978356e2f2e76"
server_key_iv: "b5eb8aa1b09d11dfa12b0e3f439124de"
server_key_version: "encryptionkey@0"
I20260923 21:01:49.711341 10725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 21:01:49.715083 10740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:49.716457 10725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:01:49.716656 10725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "42dfe49cceaf444682d33ca3598a3b68"
format_stamp: "Formatted at 2026-09-23 21:01:49 on dist-test-slave-8xn5"
server_key: "5b7e55a08270f7918c4978356e2f2e76"
server_key_iv: "b5eb8aa1b09d11dfa12b0e3f439124de"
server_key_version: "encryptionkey@0"
I20260923 21:01:49.716863 10725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:49.746558 10725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:49.751708 10725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:49.752063 10725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:49.773176 10725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35455
I20260923 21:01:49.773183 10791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35455 every 8 connection(s)
I20260923 21:01:49.774762 10725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:01:49.778825 10792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:49.781080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10725
I20260923 21:01:49.781322 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:01:49.782122 388 external_mini_cluster.cc:1428] Setting key 71547f8aa85addbba663521f4405045c
I20260923 21:01:49.792843 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68: Bootstrap starting.
I20260923 21:01:49.797008 10792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:49.798556 10792 log.cc:824] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:01:49 dist-test-slave-8xn5 krb5kdc[10712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:49.801749 10792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68: No bootstrap required, opened a new log
I20260923 21:01:49.808945 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:49.784410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.0.97.62:35455)
0923 21:01:49.785268 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:49.785281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:49.785340 (+ 59us) server_negotiation.cc:407] Connection header received
0923 21:01:49.787116 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:49.787141 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:49.787576 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:49.787940 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:49.790420 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:49.792074 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:49.793938 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:49.794967 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:49.795141 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:49.800348 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:49.800357 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:49.800377 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:49.800419 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:49.804122 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:49.805000 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:49.805011 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:49.805018 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:49.805223 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:49.805658 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:49.805665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:49.805669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:49.806321 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:49.806741 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:49.807220 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 21:01:49.807728 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":391,"threads_started":1}
I20260923 21:01:49.810122 10792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dfe49cceaf444682d33ca3598a3b68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35455 } }
I20260923 21:01:49.810653 10792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:49.810766 10792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42dfe49cceaf444682d33ca3598a3b68, State: Initialized, Role: FOLLOWER
I20260923 21:01:49.811502 10792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dfe49cceaf444682d33ca3598a3b68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35455 } }
I20260923 21:01:49.811779 10792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:01:49.811928 10792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:01:49.812150 10792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:49.814920 10792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dfe49cceaf444682d33ca3598a3b68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35455 } }
I20260923 21:01:49.815500 10792 leader_election.cc:302] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42dfe49cceaf444682d33ca3598a3b68; no voters:
I20260923 21:01:49.816129 10792 leader_election.cc:288] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:01:49.816452 10797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:49.817339 10797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [term 1 LEADER]: Becoming Leader. State: Replica: 42dfe49cceaf444682d33ca3598a3b68, State: Running, Role: LEADER
I20260923 21:01:49.817939 10797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dfe49cceaf444682d33ca3598a3b68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35455 } }
I20260923 21:01:49.818496 10792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:01:49.821173 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42dfe49cceaf444682d33ca3598a3b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dfe49cceaf444682d33ca3598a3b68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35455 } } }
I20260923 21:01:49.821574 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:49.822239 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42dfe49cceaf444682d33ca3598a3b68. Latest consensus state: current_term: 1 leader_uuid: "42dfe49cceaf444682d33ca3598a3b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42dfe49cceaf444682d33ca3598a3b68" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35455 } } }
I20260923 21:01:49.822690 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68 [sys.catalog]: This master's current role is: LEADER
I20260923 21:01:49.825169 10802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:01:49.831449 10802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:01:49.841658 10802 catalog_manager.cc:1347] Generated new cluster ID: 1371c4dd14214be2ba34342a87d89533
I20260923 21:01:49.841806 10802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:01:49.858716 10802 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:01:49.860164 10802 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:01:49.868489 10802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68: Generated new TSK 0
I20260923 21:01:49.869596 10802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:01:49.930709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35455
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:50.293529 10818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:50.293946 10818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:50.294130 10818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:50.301870 10818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:50.302053 10818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:50.302250 10818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:01:50.313026 10818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:50.314495 10818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:50.315905 10818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:50.330687 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:50.330822 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:50.330753 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:50.332242 10818 server_base.cc:1034] running on GCE node
I20260923 21:01:50.332916 10818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:50.333920 10818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:50.335147 10818 hybrid_clock.cc:648] HybridClock initialized: now 1790197310335077 us; error 77 us; skew 500 ppm
I20260923 21:01:50.335481 10818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:50 dist-test-slave-8xn5 krb5kdc[10712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:50.339010 10818 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:01:50.340647 10818 webserver.cc:460] Webserver started at http://127.0.97.1:43045/ using document root <none> and password file <none>
I20260923 21:01:50.341372 10818 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:50.341473 10818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:50.341727 10818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:50.344847 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "56a43d4244d34fc196c2011233f9ace6"
format_stamp: "Formatted at 2026-09-23 21:01:50 on dist-test-slave-8xn5"
server_key: "30d4bb4ac28e498b942a04fcdc4fa0e3"
server_key_iv: "16ce2e22be15a4f334b0a231a469916b"
server_key_version: "encryptionkey@0"
I20260923 21:01:50.345605 10818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "56a43d4244d34fc196c2011233f9ace6"
format_stamp: "Formatted at 2026-09-23 21:01:50 on dist-test-slave-8xn5"
server_key: "30d4bb4ac28e498b942a04fcdc4fa0e3"
server_key_iv: "16ce2e22be15a4f334b0a231a469916b"
server_key_version: "encryptionkey@0"
I20260923 21:01:50.351378 10818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 21:01:50.355129 10833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:50.356686 10818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:01:50.356887 10818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "56a43d4244d34fc196c2011233f9ace6"
format_stamp: "Formatted at 2026-09-23 21:01:50 on dist-test-slave-8xn5"
server_key: "30d4bb4ac28e498b942a04fcdc4fa0e3"
server_key_iv: "16ce2e22be15a4f334b0a231a469916b"
server_key_version: "encryptionkey@0"
I20260923 21:01:50.357096 10818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:50.369532 10818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:50.374083 10818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:50.374469 10818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:50.375535 10818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:50.376909 10818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:50.376982 10818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:50.377117 10818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:50.377179 10818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:50.413621 10818 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42899
I20260923 21:01:50.413668 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42899 every 8 connection(s)
I20260923 21:01:50.415068 10818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:01:50.419684 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10818
I20260923 21:01:50.419982 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:01:50.420909 388 external_mini_cluster.cc:1428] Setting key 1afe9160e8a463a1be002ed6f6658ac9
Sep 23 21:01:50 dist-test-slave-8xn5 krb5kdc[10712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:50.441308 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:50.418219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40751 (local address 127.0.97.62:35455)
0923 21:01:50.418809 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:50.418814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:50.421617 (+ 2803us) server_negotiation.cc:407] Connection header received
0923 21:01:50.423224 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:50.423229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:50.423329 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:50.423456 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:50.425296 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.426419 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.429515 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.430111 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.430234 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:50.434940 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:50.434946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:50.434949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:50.434996 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:50.437667 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:50.438747 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:50.438786 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:50.438788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:50.438915 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:50.439449 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:50.439454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:50.439456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:50.439728 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:50.439866 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:50.440860 (+ 994us) server_negotiation.cc:299] Negotiation successful
0923 21:01:50.441041 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":300,"threads_started":1}
I20260923 21:01:50.443060 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:50.419013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35455 (local address 127.0.97.1:40751)
0923 21:01:50.420159 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:50.420236 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:50.422894 (+ 2658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:50.423745 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:50.423755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:50.424336 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:50.425098 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:50.425119 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.426612 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.426621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:50.429239 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:50.429262 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.430416 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.430428 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:50.430809 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:50.431763 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:50.431789 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:50.434665 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:50.437852 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:50.437865 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:50.437889 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:50.438605 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:50.439031 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:50.439039 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:50.439045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:50.439302 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:50.439869 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:50.439881 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:50.440416 (+ 535us) client_negotiation.cc:769] Sending connection context
0923 21:01:50.440755 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 21:01:50.441386 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":417,"threads_started":1}
I20260923 21:01:50.445034 10946 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35455
I20260923 21:01:50.445436 10946 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:50.446470 10946 heartbeater.cc:511] Master 127.0.97.62:35455 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:01:50.448952 10757 ts_manager.cc:194] Registered new tserver with Master: 56a43d4244d34fc196c2011233f9ace6 (127.0.97.1:42899)
I20260923 21:01:50.450651 10757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:01:50.483654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35455
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:50.826195 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:50.826556 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:50.826658 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:50.834178 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:50.834304 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:50.834514 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:01:50.844661 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:50.845958 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:50.847421 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:50.861375 10957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:50.861546 10960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:50.861755 10958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:50.863308 10952 server_base.cc:1034] running on GCE node
I20260923 21:01:50.864054 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:50.865145 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:50.866467 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790197310866421 us; error 58 us; skew 500 ppm
I20260923 21:01:50.866833 10952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:50 dist-test-slave-8xn5 krb5kdc[10712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:50.874686 10952 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:01:50.876410 10952 webserver.cc:460] Webserver started at http://127.0.97.2:40589/ using document root <none> and password file <none>
I20260923 21:01:50.877203 10952 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:50.877352 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:50.877664 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:50.880777 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "1ca9ac80e7174eb3adef17e691caa996"
format_stamp: "Formatted at 2026-09-23 21:01:50 on dist-test-slave-8xn5"
server_key: "91f622445bd4914efbb99a92e3c0b65f"
server_key_iv: "3c08f43eab49c2437791bbcb32737d5f"
server_key_version: "encryptionkey@0"
I20260923 21:01:50.881552 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1ca9ac80e7174eb3adef17e691caa996"
format_stamp: "Formatted at 2026-09-23 21:01:50 on dist-test-slave-8xn5"
server_key: "91f622445bd4914efbb99a92e3c0b65f"
server_key_iv: "3c08f43eab49c2437791bbcb32737d5f"
server_key_version: "encryptionkey@0"
I20260923 21:01:50.887312 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 21:01:50.891414 10967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:50.893853 10952 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260923 21:01:50.894110 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "1ca9ac80e7174eb3adef17e691caa996"
format_stamp: "Formatted at 2026-09-23 21:01:50 on dist-test-slave-8xn5"
server_key: "91f622445bd4914efbb99a92e3c0b65f"
server_key_iv: "3c08f43eab49c2437791bbcb32737d5f"
server_key_version: "encryptionkey@0"
I20260923 21:01:50.894428 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:50.918778 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:50.925205 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:50.925580 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:50.927117 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:50.929332 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:50.929443 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:50.929610 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:50.929693 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:50.966663 10952 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43207
I20260923 21:01:50.966684 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43207 every 8 connection(s)
I20260923 21:01:50.968113 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:01:50.968497 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 10952
I20260923 21:01:50.968699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:01:50.969256 388 external_mini_cluster.cc:1428] Setting key bbdc086e71febb64d193b0b8c9ea9c75
Sep 23 21:01:50 dist-test-slave-8xn5 krb5kdc[10712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:50.992417 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:50.971290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35979 (local address 127.0.97.62:35455)
0923 21:01:50.971760 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:50.971767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:50.973333 (+ 1566us) server_negotiation.cc:407] Connection header received
0923 21:01:50.975417 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:50.975423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:50.975523 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:50.975648 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:50.977290 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.978092 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.980776 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.981197 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.981304 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:50.986058 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:50.986063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:50.986067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:50.986101 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:50.988854 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:50.989698 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:50.989709 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:50.989711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:50.989889 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:50.990387 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:50.990391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:50.990393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:50.990676 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:50.990831 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:50.991927 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0923 21:01:50.992121 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":640,"threads_started":1}
I20260923 21:01:50.994505 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:50.972194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35455 (local address 127.0.97.2:35979)
0923 21:01:50.973121 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:50.973179 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:50.975115 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:50.975918 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:50.975928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:50.976476 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:50.977092 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:50.977110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.978284 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.978289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:50.980575 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:50.980595 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:50.981477 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:50.981487 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:50.981884 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:50.982792 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:50.982837 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:50.985758 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:50.989033 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:50.989043 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:50.989071 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:50.989545 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:50.990012 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:50.990017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:50.990021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:50.990224 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:50.990850 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:50.990863 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:50.991422 (+ 559us) client_negotiation.cc:769] Sending connection context
0923 21:01:50.991844 (+ 422us) client_negotiation.cc:240] Negotiation successful
0923 21:01:50.992753 (+ 909us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":403,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:01:50.996245 11080 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35455
I20260923 21:01:50.996542 11080 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:50.997367 11080 heartbeater.cc:511] Master 127.0.97.62:35455 requested a full tablet report, sending...
I20260923 21:01:50.999058 10757 ts_manager.cc:194] Registered new tserver with Master: 1ca9ac80e7174eb3adef17e691caa996 (127.0.97.2:43207)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:01:51.000243 10757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:01:51.031278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35455
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:01:51.370288 11086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:01:51.370733 11086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:01:51.370904 11086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:01:51.378813 11086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:01:51.378985 11086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:01:51.379213 11086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:01:51.389417 11086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:01:51.391131 11086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:01:51.392560 11086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:01:51.408162 11091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:51.408118 11092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:01:51.408072 11094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:01:51.409111 11086 server_base.cc:1034] running on GCE node
I20260923 21:01:51.409869 11086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:01:51.411060 11086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:01:51.412329 11086 hybrid_clock.cc:648] HybridClock initialized: now 1790197311412275 us; error 42 us; skew 500 ppm
I20260923 21:01:51.412714 11086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:01:51 dist-test-slave-8xn5 krb5kdc[10712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:01:51.416525 11086 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:01:51.418900 11086 webserver.cc:460] Webserver started at http://127.0.97.3:44925/ using document root <none> and password file <none>
I20260923 21:01:51.419814 11086 fs_manager.cc:359] Metadata directory not provided
I20260923 21:01:51.420004 11086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:01:51.420410 11086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:01:51.423869 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8"
format_stamp: "Formatted at 2026-09-23 21:01:51 on dist-test-slave-8xn5"
server_key: "c101890740612aabf3985ca71620f8bc"
server_key_iv: "b8de641bfa453eb514ae7ae6dfa034f4"
server_key_version: "encryptionkey@0"
I20260923 21:01:51.424739 11086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8"
format_stamp: "Formatted at 2026-09-23 21:01:51 on dist-test-slave-8xn5"
server_key: "c101890740612aabf3985ca71620f8bc"
server_key_iv: "b8de641bfa453eb514ae7ae6dfa034f4"
server_key_version: "encryptionkey@0"
I20260923 21:01:51.431125 11086 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260923 21:01:51.436545 11101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:51.438275 11086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:01:51.438632 11086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8"
format_stamp: "Formatted at 2026-09-23 21:01:51 on dist-test-slave-8xn5"
server_key: "c101890740612aabf3985ca71620f8bc"
server_key_iv: "b8de641bfa453eb514ae7ae6dfa034f4"
server_key_version: "encryptionkey@0"
I20260923 21:01:51.438889 11086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:01:51.454566 10946 heartbeater.cc:503] Master 127.0.97.62:35455 was elected leader, sending a full tablet report...
I20260923 21:01:51.470831 11086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:01:51.475534 11086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:01:51.475795 11086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:01:51.476740 11086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:01:51.478119 11086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:01:51.478191 11086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:51.478276 11086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:01:51.478320 11086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:01:51.513290 11086 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39271
I20260923 21:01:51.513350 11213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39271 every 8 connection(s)
I20260923 21:01:51.514729 11086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:01:51.515483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 11086
I20260923 21:01:51.515681 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:01:51.516211 388 external_mini_cluster.cc:1428] Setting key eb2ba32d6a4b0081d9b2768d3c0ad296
Sep 23 21:01:51 dist-test-slave-8xn5 krb5kdc[10712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:01:51.543673 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.517604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48667 (local address 127.0.97.62:35455)
0923 21:01:51.520688 (+ 3084us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.520693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.520710 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:01:51.522513 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.522517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.522603 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.522698 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:51.524612 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.525534 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.527615 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.528027 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.528134 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.537163 (+ 9029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:51.537168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:51.537172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:51.537212 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:51.539996 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:51.540937 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:51.540942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:51.540944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:51.541077 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:51.541581 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:51.541585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:51.541587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:51.541904 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:51.542025 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.542979 (+ 954us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.543217 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2989,"thread_start_us":2868,"threads_started":1}
I20260923 21:01:51.544538 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.518596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35455 (local address 127.0.97.3:48667)
0923 21:01:51.519898 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:51.519961 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:51.522089 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:51.522959 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:51.522968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:51.523716 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:01:51.524399 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.524421 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.525753 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.525757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.527432 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.527447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.528240 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.528245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.528544 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.534205 (+ 5661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:01:51.534233 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:01:51.536918 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:01:51.540178 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:51.540187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:51.540203 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:51.540761 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:51.541180 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:01:51.541184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:01:51.541186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:01:51.541455 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:01:51.542027 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:01:51.542033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:01:51.542530 (+ 497us) client_negotiation.cc:769] Sending connection context
0923 21:01:51.542789 (+ 259us) client_negotiation.cc:240] Negotiation successful
0923 21:01:51.543209 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":426,"threads_started":1}
I20260923 21:01:51.546263 11214 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35455
I20260923 21:01:51.546628 11214 heartbeater.cc:464] Registering TS with master...
I20260923 21:01:51.547377 11214 heartbeater.cc:511] Master 127.0.97.62:35455 requested a full tablet report, sending...
I20260923 21:01:51.548916 10756 ts_manager.cc:194] Registered new tserver with Master: dd4eb541cd0b47ed8709dd29a4ec70d8 (127.0.97.3:39271)
I20260923 21:01:51.549906 10756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48667
I20260923 21:01:51.559187 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:01:51.577510 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.564252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.0.97.62:35455)
0923 21:01:51.564485 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.564490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.564882 (+ 392us) server_negotiation.cc:407] Connection header received
0923 21:01:51.565023 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.565027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.565094 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.565191 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:51.566973 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.567819 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.569076 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.569534 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.569650 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.571989 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:51.571993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:51.571997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:51.572030 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:51.574824 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:51.575653 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:51.575658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:51.575660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:51.575771 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:51.576112 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:51.576116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:51.576117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:51.576434 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:51.576607 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.577076 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.577272 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260923 21:01:51.582641 10756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:01:51.590170 10756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:01:51.625895 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.603911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39271 (local address 127.0.0.1:46162)
0923 21:01:51.614097 (+ 10186us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:51.614153 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:51.614372 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:51.615698 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:51.615707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:51.615759 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:51.616198 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.616212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.619495 (+ 3283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.619510 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.623019 (+ 3509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.623034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.624643 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.624649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.624864 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.624928 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 21:01:51.625090 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 21:01:51.625324 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9922,"mutex_wait_us":40,"thread_start_us":10813,"threads_started":1}
I20260923 21:01:51.626674 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.604144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.0.1:40332)
0923 21:01:51.615047 (+ 10903us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:51.615068 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:51.615185 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:51.616142 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:51.616152 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:51.616183 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:51.616639 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.616656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.619321 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.619329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.623219 (+ 3890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.623231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.625093 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.625101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.625371 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.625408 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 21:01:51.625527 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 21:01:51.625727 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10748,"mutex_wait_us":26,"thread_start_us":9522,"threads_started":1}
I20260923 21:01:51.626860 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.604035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.0.97.3:39271)
0923 21:01:51.614967 (+ 10932us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.614977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.615014 (+ 37us) server_negotiation.cc:407] Connection header received
0923 21:01:51.615129 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.615139 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.615438 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.616449 (+ 1011us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:51.616831 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.619280 (+ 2449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.623213 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.624411 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.624630 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.624914 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.626228 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.626462 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10741,"thread_start_us":10549,"threads_started":1}
I20260923 21:01:51.627345 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.604758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40332 (local address 127.0.97.1:42899)
0923 21:01:51.613622 (+ 8864us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.613636 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.615325 (+ 1689us) server_negotiation.cc:407] Connection header received
0923 21:01:51.615429 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.615436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.615688 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.615886 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:51.616871 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.619110 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.623376 (+ 4266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.624898 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.625021 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.625266 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.625622 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.625812 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8601,"thread_start_us":8387,"threads_started":1}
I20260923 21:01:51.630982 10881 tablet_service.cc:1467] Processing CreateTablet for tablet 709b68b401c249e7b983f7ee2695e17d (DEFAULT_TABLE table=test-table [id=6a0e5141e67543f5b3e5353976bd945c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:51.631729 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.616963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43207 (local address 127.0.0.1:55080)
0923 21:01:51.621760 (+ 4797us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:51.621806 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:51.621938 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:51.622670 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:51.622675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:51.622707 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:51.623400 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.623416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.625273 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.625292 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.628676 (+ 3384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.628693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.630883 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.630889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.631069 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.631104 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:01:51.631205 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 21:01:51.631353 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4638,"mutex_wait_us":74,"thread_start_us":4438,"threads_started":1}
I20260923 21:01:51.631160 11149 tablet_service.cc:1467] Processing CreateTablet for tablet 709b68b401c249e7b983f7ee2695e17d (DEFAULT_TABLE table=test-table [id=6a0e5141e67543f5b3e5353976bd945c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:51.631825 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.615441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.0.97.2:43207)
0923 21:01:51.620364 (+ 4923us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.620373 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.621925 (+ 1552us) server_negotiation.cc:407] Connection header received
0923 21:01:51.622158 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.622167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.622446 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.622652 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:51.623587 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.625108 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.628871 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.630648 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.630797 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.630995 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.631294 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.631486 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4747,"thread_start_us":4604,"threads_started":1}
I20260923 21:01:51.632611 11149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709b68b401c249e7b983f7ee2695e17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:51.633641 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709b68b401c249e7b983f7ee2695e17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:51.635582 11015 tablet_service.cc:1467] Processing CreateTablet for tablet 709b68b401c249e7b983f7ee2695e17d (DEFAULT_TABLE table=test-table [id=6a0e5141e67543f5b3e5353976bd945c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:01:51.636916 11015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 709b68b401c249e7b983f7ee2695e17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:01:51.645855 11233 tablet_bootstrap.cc:492] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Bootstrap starting.
I20260923 21:01:51.649232 11234 tablet_bootstrap.cc:492] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Bootstrap starting.
I20260923 21:01:51.649843 11233 tablet_bootstrap.cc:654] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:51.651464 11235 tablet_bootstrap.cc:492] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Bootstrap starting.
I20260923 21:01:51.651496 11233 log.cc:824] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:51.653147 11234 tablet_bootstrap.cc:654] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:51.654639 11234 log.cc:824] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:51.655262 11233 tablet_bootstrap.cc:492] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: No bootstrap required, opened a new log
I20260923 21:01:51.655517 11235 tablet_bootstrap.cc:654] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Neither blocks nor log segments found. Creating new log.
I20260923 21:01:51.655572 11233 ts_tablet_manager.cc:1397] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20260923 21:01:51.656790 11235 log.cc:824] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Log is configured to *not* fsync() on all Append() calls
I20260923 21:01:51.658061 11234 tablet_bootstrap.cc:492] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: No bootstrap required, opened a new log
I20260923 21:01:51.658417 11234 ts_tablet_manager.cc:1397] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.007s
I20260923 21:01:51.660189 11235 tablet_bootstrap.cc:492] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: No bootstrap required, opened a new log
I20260923 21:01:51.660523 11235 ts_tablet_manager.cc:1397] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 21:01:51.664357 11233 raft_consensus.cc:348] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.664789 11233 raft_consensus.cc:374] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:51.664889 11233 raft_consensus.cc:729] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4eb541cd0b47ed8709dd29a4ec70d8, State: Initialized, Role: FOLLOWER
I20260923 21:01:51.665598 11233 consensus_queue.cc:260] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.666771 11214 heartbeater.cc:503] Master 127.0.97.62:35455 was elected leader, sending a full tablet report...
I20260923 21:01:51.667635 11233 ts_tablet_manager.cc:1428] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260923 21:01:51.670447 11234 raft_consensus.cc:348] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.670969 11234 raft_consensus.cc:374] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:51.671074 11234 raft_consensus.cc:729] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a43d4244d34fc196c2011233f9ace6, State: Initialized, Role: FOLLOWER
I20260923 21:01:51.671779 11234 consensus_queue.cc:260] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.673969 11234 ts_tablet_manager.cc:1428] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260923 21:01:51.674113 11235 raft_consensus.cc:348] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.674641 11235 raft_consensus.cc:374] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:01:51.674764 11235 raft_consensus.cc:729] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca9ac80e7174eb3adef17e691caa996, State: Initialized, Role: FOLLOWER
I20260923 21:01:51.675494 11235 consensus_queue.cc:260] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.676757 11080 heartbeater.cc:503] Master 127.0.97.62:35455 was elected leader, sending a full tablet report...
I20260923 21:01:51.677429 11235 ts_tablet_manager.cc:1428] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
W20260923 21:01:51.720664 11081 tablet.cc:2369] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:51.749639 11239 raft_consensus.cc:482] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:01:51.749998 11239 raft_consensus.cc:504] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.752079 11239 leader_election.cc:288] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56a43d4244d34fc196c2011233f9ace6 (127.0.97.1:42899), 1ca9ac80e7174eb3adef17e691caa996 (127.0.97.2:43207)
I20260923 21:01:51.759557 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.752484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37279 (local address 127.0.97.1:42899)
0923 21:01:51.752866 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.752873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.752896 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:01:51.753051 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.753059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.753158 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.753270 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:51.753815 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.754917 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.757332 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.758611 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.758793 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.758907 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.759197 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.759307 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260923 21:01:51.759719 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.752434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42899 (local address 127.0.97.3:37279)
0923 21:01:51.752707 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:51.752730 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:51.752879 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:51.753330 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:51.753335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:51.753379 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:51.753626 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.753636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.755060 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.755067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.757156 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.757172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.758798 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.758804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.759023 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.759062 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 21:01:51.759171 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 21:01:51.759326 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1}
I20260923 21:01:51.760803 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.752698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43207 (local address 127.0.97.3:57339)
0923 21:01:51.753611 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0923 21:01:51.753627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:01:51.753777 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:01:51.754191 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:01:51.754195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:01:51.754223 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:01:51.754590 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.754602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.755717 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.755723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.757832 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:01:51.757846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.759612 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.759621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:01:51.760080 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.760120 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 21:01:51.760284 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 21:01:51.760431 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":28,"thread_start_us":571,"threads_started":1}
I20260923 21:01:51.760885 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.752846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57339 (local address 127.0.97.2:43207)
0923 21:01:51.753085 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.753095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.753724 (+ 629us) server_negotiation.cc:407] Connection header received
0923 21:01:51.753933 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.753938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.754048 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.754150 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:01:51.754772 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.755547 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.758030 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.759371 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.759516 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.759643 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.760353 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.760611 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 21:01:51.761052 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709b68b401c249e7b983f7ee2695e17d" candidate_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a43d4244d34fc196c2011233f9ace6" is_pre_election: true
I20260923 21:01:51.761703 10901 raft_consensus.cc:2454] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4eb541cd0b47ed8709dd29a4ec70d8 in term 0.
I20260923 21:01:51.762033 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709b68b401c249e7b983f7ee2695e17d" candidate_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca9ac80e7174eb3adef17e691caa996" is_pre_election: true
I20260923 21:01:51.762567 11102 leader_election.cc:302] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56a43d4244d34fc196c2011233f9ace6, dd4eb541cd0b47ed8709dd29a4ec70d8; no voters:
I20260923 21:01:51.762633 11035 raft_consensus.cc:2454] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd4eb541cd0b47ed8709dd29a4ec70d8 in term 0.
I20260923 21:01:51.763049 11239 raft_consensus.cc:2781] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:01:51.763231 11239 raft_consensus.cc:482] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:01:51.763370 11239 raft_consensus.cc:3037] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:51.765830 11239 raft_consensus.cc:504] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.766615 11239 leader_election.cc:288] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [CANDIDATE]: Term 1 election: Requested vote from peers 56a43d4244d34fc196c2011233f9ace6 (127.0.97.1:42899), 1ca9ac80e7174eb3adef17e691caa996 (127.0.97.2:43207)
W20260923 21:01:51.766664 11215 tablet.cc:2369] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:01:51.767103 11035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709b68b401c249e7b983f7ee2695e17d" candidate_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca9ac80e7174eb3adef17e691caa996"
I20260923 21:01:51.767144 10901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "709b68b401c249e7b983f7ee2695e17d" candidate_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a43d4244d34fc196c2011233f9ace6"
I20260923 21:01:51.767375 11035 raft_consensus.cc:3037] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:51.767400 10901 raft_consensus.cc:3037] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:01:51.769543 11035 raft_consensus.cc:2454] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4eb541cd0b47ed8709dd29a4ec70d8 in term 1.
I20260923 21:01:51.769783 10901 raft_consensus.cc:2454] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd4eb541cd0b47ed8709dd29a4ec70d8 in term 1.
I20260923 21:01:51.770015 11103 leader_election.cc:302] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca9ac80e7174eb3adef17e691caa996, dd4eb541cd0b47ed8709dd29a4ec70d8; no voters:
I20260923 21:01:51.770403 11239 raft_consensus.cc:2781] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:01:51.770900 11239 raft_consensus.cc:686] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 1 LEADER]: Becoming Leader. State: Replica: dd4eb541cd0b47ed8709dd29a4ec70d8, State: Running, Role: LEADER
I20260923 21:01:51.771411 11239 consensus_queue.cc:237] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } }
I20260923 21:01:51.776726 10756 catalog_manager.cc:5495] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd4eb541cd0b47ed8709dd29a4ec70d8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd4eb541cd0b47ed8709dd29a4ec70d8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 } health_report { overall_health: UNKNOWN } } }
I20260923 21:01:51.808640 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.803174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.0.97.3:39271)
0923 21:01:51.803403 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.803408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.803534 (+ 126us) server_negotiation.cc:407] Connection header received
0923 21:01:51.803723 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.803727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.803806 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.803933 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:01:51.804320 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.804992 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.806487 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.806865 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.806958 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.807609 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:01:51.808112 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:01:51.808229 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.808375 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.808440 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:01:51.817823 10901 raft_consensus.cc:1261] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 1 FOLLOWER]: Refusing update from remote peer dd4eb541cd0b47ed8709dd29a4ec70d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:51.817865 11035 raft_consensus.cc:1261] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 1 FOLLOWER]: Refusing update from remote peer dd4eb541cd0b47ed8709dd29a4ec70d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:01:51.819105 11239 consensus_queue.cc:1035] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca9ac80e7174eb3adef17e691caa996" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:51.819598 11243 consensus_queue.cc:1035] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a43d4244d34fc196c2011233f9ace6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:01:51.833509 11248 mvcc.cc:205] Tried to move back new op lower bound from 7332648189194035200 to 7332648189019574272. Current Snapshot: MvccSnapshot[applied={T|T < 7332648189194035200}]
I20260923 21:01:51.845916 10756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:51.846159 10756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:01:51.850441 10756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42dfe49cceaf444682d33ca3598a3b68: Sending DeleteTablet for 3 replicas of tablet 709b68b401c249e7b983f7ee2695e17d
I20260923 21:01:51.851843 10881 tablet_service.cc:1514] Processing DeleteTablet for tablet 709b68b401c249e7b983f7ee2695e17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40332
I20260923 21:01:51.852028 11015 tablet_service.cc:1514] Processing DeleteTablet for tablet 709b68b401c249e7b983f7ee2695e17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55080
I20260923 21:01:51.852671 11149 tablet_service.cc:1514] Processing DeleteTablet for tablet 709b68b401c249e7b983f7ee2695e17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:01:51 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46162
I20260923 21:01:51.853209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10818
I20260923 21:01:51.867219 11258 tablet_replica.cc:330] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: stopping tablet replica
I20260923 21:01:51.867506 11256 tablet_replica.cc:330] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: stopping tablet replica
I20260923 21:01:51.867727 11257 tablet_replica.cc:330] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: stopping tablet replica
I20260923 21:01:51.867743 11258 raft_consensus.cc:2229] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:01:51.868111 11256 raft_consensus.cc:2229] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:51.868223 11257 raft_consensus.cc:2229] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:01:51.868418 11258 raft_consensus.cc:2258] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:51.868803 11257 raft_consensus.cc:2258] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:51.869038 11256 raft_consensus.cc:2258] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:01:51.871206 11258 ts_tablet_manager.cc:1905] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:51.871802 11256 ts_tablet_manager.cc:1905] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:51.876292 11257 ts_tablet_manager.cc:1905] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:01:51.878566 11258 ts_tablet_manager.cc:1918] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:51.878798 11258 log.cc:1196] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/709b68b401c249e7b983f7ee2695e17d
I20260923 21:01:51.878829 11256 ts_tablet_manager.cc:1918] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:51.878994 11256 log.cc:1196] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/709b68b401c249e7b983f7ee2695e17d
I20260923 21:01:51.879424 11258 ts_tablet_manager.cc:1939] T 709b68b401c249e7b983f7ee2695e17d P dd4eb541cd0b47ed8709dd29a4ec70d8: Deleting consensus metadata
I20260923 21:01:51.879535 11256 ts_tablet_manager.cc:1939] T 709b68b401c249e7b983f7ee2695e17d P 56a43d4244d34fc196c2011233f9ace6: Deleting consensus metadata
I20260923 21:01:51.881422 10741 catalog_manager.cc:4841] TS 56a43d4244d34fc196c2011233f9ace6 (127.0.97.1:42899): tablet 709b68b401c249e7b983f7ee2695e17d (table test-table [id=6a0e5141e67543f5b3e5353976bd945c]) successfully deleted
I20260923 21:01:51.881494 10742 catalog_manager.cc:4841] TS dd4eb541cd0b47ed8709dd29a4ec70d8 (127.0.97.3:39271): tablet 709b68b401c249e7b983f7ee2695e17d (table test-table [id=6a0e5141e67543f5b3e5353976bd945c]) successfully deleted
Sep 23 21:01:51 dist-test-slave-8xn5 krb5kdc[10712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:01:51.887614 11257 ts_tablet_manager.cc:1918] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:01:51.887841 11257 log.cc:1196] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/709b68b401c249e7b983f7ee2695e17d
I20260923 21:01:51.888342 11257 ts_tablet_manager.cc:1939] T 709b68b401c249e7b983f7ee2695e17d P 1ca9ac80e7174eb3adef17e691caa996: Deleting consensus metadata
I20260923 21:01:51.890291 10742 catalog_manager.cc:4841] TS 1ca9ac80e7174eb3adef17e691caa996 (127.0.97.2:43207): tablet 709b68b401c249e7b983f7ee2695e17d (table test-table [id=6a0e5141e67543f5b3e5353976bd945c]) successfully deleted
I20260923 21:01:51.893862 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:51.854577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.0.97.1:42899)
0923 21:01:51.854782 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:51.854790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:51.865020 (+ 10230us) server_negotiation.cc:407] Connection header received
0923 21:01:51.866407 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:51.866417 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:51.866583 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:51.866701 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:51.868995 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.870302 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.879642 (+ 9340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:51.880493 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:51.880613 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:51.886852 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:51.886872 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:51.886893 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:51.886941 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:51.890864 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:51.891609 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:51.891619 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:51.891623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:51.891785 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:51.892191 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:51.892197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:51.892202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:51.892654 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:51.892943 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:51.893232 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 21:01:51.893464 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:01:51.894675 10941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:51.986204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10818
I20260923 21:01:52.005267 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10952
Sep 23 21:01:52 dist-test-slave-8xn5 krb5kdc[10712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:01:52.020884 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:52.006082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.0.97.2:43207)
0923 21:01:52.006253 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:52.006258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:52.006413 (+ 155us) server_negotiation.cc:407] Connection header received
0923 21:01:52.006618 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:52.006624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:52.006711 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:52.006828 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:52.008041 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:52.008774 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:52.009987 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:52.010453 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:52.010589 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:52.014672 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:52.014687 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:52.014704 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:52.014750 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:52.017642 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:52.018721 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:52.018749 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:52.018751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:52.018875 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:52.019310 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:52.019313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:52.019315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:52.019691 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:52.020000 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:52.020318 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 21:01:52.020478 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:01:52.021569 11075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:52.135555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10952
I20260923 21:01:52.152808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 11086
Sep 23 21:01:52 dist-test-slave-8xn5 krb5kdc[10712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:01:52.170017 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:01:52.153626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.0.97.3:39271)
0923 21:01:52.153790 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 21:01:52.153796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:01:52.153886 (+ 90us) server_negotiation.cc:407] Connection header received
0923 21:01:52.154090 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:01:52.154095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:01:52.154190 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:01:52.154318 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:01:52.155609 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:52.156428 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:52.157730 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:01:52.158155 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:01:52.158301 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:01:52.163414 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:01:52.163427 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:01:52.163443 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:01:52.163489 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:01:52.166910 (+ 3421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:52.167832 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:52.167841 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:52.167843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:52.167986 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:01:52.168395 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:01:52.168399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:01:52.168401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:01:52.168753 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:01:52.169106 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0923 21:01:52.169422 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 21:01:52.169645 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:01:52.170984 11209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:52.296816 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 11086
I20260923 21:01:52.318444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 10725
I20260923 21:01:52.319788 10787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:01:52.431380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 10725
2026-09-23T21:01:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4502 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:01:52 dist-test-slave-8xn5 krb5kdc[11266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:01:52 dist-test-slave-8xn5 krb5kdc[11266](info): set up 2 sockets
Sep 23 21:01:52 dist-test-slave-8xn5 krb5kdc[11266](info): commencing operation
krb5kdc: starting...
W20260923 21:01:53.551852 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:01:53 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:01:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260923 21:01:53.905496 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDsUdkt/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/postgres -l logfile start
2026-09-23 21:01:56.797 UTC [11296] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 21:01:56.797 UTC [11296] LOG: listening on IPv4 address "127.0.97.20", port 48343
2026-09-23 21:01:56.802 UTC [11296] LOG: listening on Unix socket "/tmp/.s.PGSQL.48343"
2026-09-23 21:01:56.829 UTC [11299] LOG: database system was shut down at 2026-09-23 21:01:54 UTC
2026-09-23 21:01:56.834 UTC [11296] LOG: database system is ready to accept connections
I20260923 21:01:57.814911 388 mini_postgres.cc:96] Postgres bound to 48343
2026-09-23 21:01:57.833 UTC [11307] FATAL: database "slave" does not exist
127.0.97.20:48343 - accepting connections
I20260923 21:01:57.834025 388 mini_ranger.cc:161] Starting Ranger...
I20260923 21:01:57.863160 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 21:01:58.468407 388 mini_ranger.cc:87] Created ranger Postgres database
I20260923 21:01:58.468621 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-admin
2026-09-23 21:01:58,796 [I] DB FLAVOR :POSTGRES
2026-09-23 21:01:58,796 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 21:01:58,796 [I] Checking connection..
2026-09-23 21:01:58,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 21:01:59,078 [I] Checking connection passed.
2026-09-23 21:01:59,078 [I] --------- Verifying version history table ---------
2026-09-23 21:01:59,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 21:01:59,341 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 21:01:59,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 21:01:59,599 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 21:01:59,599 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 21:01:59,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 21:01:59.866 UTC [11371] WARNING: there is no transaction in progress
2026-09-23 21:01:59,873 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 21:01:59,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 21:02:00,162 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 21:02:00,162 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 21:02:00,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 21:02:00,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 21:02:00,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8xn5.c.gcp-upstream.internal','N') ;"
2026-09-23 21:02:00,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 21:02:01,235 [I] Table x_portal_user does not exist in database ranger
2026-09-23 21:02:01,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 21:02:01,513 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 21:02:01,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 21:02:01,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 21:02:02,030 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 21:02:02,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 21:02:03.150 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.169 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.189 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.207 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.225 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.248 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.316 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.325 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.344 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.357 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.370 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.386 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.398 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.407 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.417 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:03.426 UTC [11479] WARNING: there is no transaction in progress
2026-09-23 21:02:04,277 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 21:02:04,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 21:02:04,553 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 21:02:04,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 21:02:04,813 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 21:02:04,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 21:02:05,068 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 21:02:05,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 21:02:05,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 21:02:05,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8xn5.c.gcp-upstream.internal';"
2026-09-23 21:02:05,878 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 21:02:05,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 21:02:06,166 [I] DB_PATCHES have already been applied
I20260923 21:02:06.173364 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDsUdkt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 21:02:06.173466 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260923 21:02:06.189818 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:37589
Sep 23, 2026 9:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 9:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 9:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 9:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 9:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 9:02:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 9:02:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 21:02:07.341114 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.507s user 0.004s sys 0.041s
Sep 23, 2026 9:02:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 9:02:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 9:02:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 9:02:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 9:02:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 21:02:30.712590 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 21:02:30 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 21:02:30.886431 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 21:02:30.911278 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 21:02:31.902673 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 21:02:31.902850 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms
I20260923 21:02:31.902992 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48343
2026-09-23 21:02:32,345 [I] DB FLAVOR :POSTGRES
2026-09-23 21:02:32,345 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 21:02:32,346 [I] Checking connection
2026-09-23 21:02:32,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 21:02:32,605 [I] connection success
2026-09-23 21:02:32,605 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 21:02:32,606 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 21:02:32,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 21:02:32,880 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 21:02:32,880 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 21:02:32,881 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 21:02:32,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 21:02:33,162 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 21:02:33.648566 388 mini_ranger_kms.cc:310] Created kms service
I20260923 21:02:33.882252 388 mini_ranger_kms.cc:326] Created kudu user
I20260923 21:02:33.932159 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 21:02:33 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:02:34.371836 388 mini_ranger_kms.cc:384] Added ranger policy
I20260923 21:02:34.372195 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 21:02:34.372238 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260923 21:02:34.395784 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11660
I20260923 21:02:34.395992 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:55701
Sep 23, 2026 9:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 9:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 9:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 9:02:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 9:02:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 9:02:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 21:02:36 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790197356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:02:36.779393 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.893s user 0.133s sys 0.329s
I20260923 21:02:36.779677 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 21:02:36.779912 388 mini_ranger_kms.cc:401] 127.0.97.20:55701/kms/v1/keys
Sep 23 21:02:37 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 21:02:37 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197356, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 21:02:39 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:02:39.847904 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34109
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34109
--ranger_config_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55701/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:02:40.204507 11710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:02:40.204831 11710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:02:40.204919 11710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:02:40.214916 11710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:02:40.215068 11710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:02:40.215155 11710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:02:40.215242 11710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:02:40.215335 11710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:02:40.226648 11710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:55701/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34109
--ranger_config_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:02:40.227917 11710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:02:40.229149 11710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:02:40.242002 11715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:02:40.242072 11718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:02:40.242096 11716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:02:40.243276 11710 server_base.cc:1034] running on GCE node
I20260923 21:02:40.244099 11710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:02:40.245679 11710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:02:40.246903 11710 hybrid_clock.cc:648] HybridClock initialized: now 1790197360246847 us; error 48 us; skew 500 ppm
I20260923 21:02:40.247330 11710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:02:40 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:02:40.258246 11710 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:02:40.260272 11710 webserver.cc:460] Webserver started at http://127.0.97.62:34543/ using document root <none> and password file <none>
I20260923 21:02:40.261148 11710 fs_manager.cc:359] Metadata directory not provided
I20260923 21:02:40.261322 11710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:02:40.261696 11710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:02:40 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:02:40.316339 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2"
format_stamp: "Formatted at 2026-09-23 21:02:40 on dist-test-slave-8xn5"
server_key: "42257cd9e868d6fcf87635ad12f94543"
server_key_iv: "ca197f5ca541fbd0fabd1e7b2ddb35e5"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:40.317289 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2"
format_stamp: "Formatted at 2026-09-23 21:02:40 on dist-test-slave-8xn5"
server_key: "42257cd9e868d6fcf87635ad12f94543"
server_key_iv: "ca197f5ca541fbd0fabd1e7b2ddb35e5"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:40.323410 11710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 21:02:40.345120 11725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:40.347025 11710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:02:40.347259 11710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2"
format_stamp: "Formatted at 2026-09-23 21:02:40 on dist-test-slave-8xn5"
server_key: "42257cd9e868d6fcf87635ad12f94543"
server_key_iv: "ca197f5ca541fbd0fabd1e7b2ddb35e5"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:40.347520 11710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:02:40.364297 11710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:02:40.368700 11710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:02:40.369436 11710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:02:40.391530 11710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34109
I20260923 21:02:40.391547 11776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34109 every 8 connection(s)
I20260923 21:02:40.392937 11710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:02:40.396905 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:02:40.398084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 11710
I20260923 21:02:40.398310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:02:40.399191 388 external_mini_cluster.cc:1428] Setting key 680f56f3c242fcd6d25c1f8738d36f69
I20260923 21:02:40.411741 11777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2: Bootstrap starting.
Sep 23 21:02:40 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:02:40.415588 11777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2: Neither blocks nor log segments found. Creating new log.
I20260923 21:02:40.417646 11777 log.cc:824] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2: Log is configured to *not* fsync() on all Append() calls
I20260923 21:02:40.420909 11777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2: No bootstrap required, opened a new log
I20260923 21:02:40.423453 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:40.401728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.97.62:34109)
0923 21:02:40.402594 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:40.402605 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:40.402645 (+ 40us) server_negotiation.cc:407] Connection header received
0923 21:02:40.404343 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:40.404367 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:40.404803 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:40.405249 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:40.406609 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:40.407906 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:40.409219 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:40.409950 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:40.410144 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:40.414535 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:40.414543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:40.414559 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:40.414601 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:40.418684 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:40.419445 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:40.419458 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:40.419464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:40.419630 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:40.420032 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:40.420039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:40.420042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:40.420570 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:40.420864 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:40.421449 (+ 585us) server_negotiation.cc:299] Negotiation successful
0923 21:02:40.421912 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":339,"threads_started":1}
I20260923 21:02:40.431218 11777 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34109 } }
I20260923 21:02:40.431595 11777 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:02:40.431684 11777 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdbdd396e9e4ef1a64b7ad1f9130aa2, State: Initialized, Role: FOLLOWER
I20260923 21:02:40.432286 11777 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34109 } }
I20260923 21:02:40.432508 11777 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:02:40.432608 11777 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:02:40.432751 11777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:02:40.435735 11777 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34109 } }
I20260923 21:02:40.436225 11777 leader_election.cc:302] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbdbdd396e9e4ef1a64b7ad1f9130aa2; no voters:
I20260923 21:02:40.436759 11777 leader_election.cc:288] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:02:40.436934 11782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:02:40.437665 11782 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [term 1 LEADER]: Becoming Leader. State: Replica: cbdbdd396e9e4ef1a64b7ad1f9130aa2, State: Running, Role: LEADER
I20260923 21:02:40.438283 11782 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34109 } }
I20260923 21:02:40.439150 11777 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:02:40.441524 11784 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbdbdd396e9e4ef1a64b7ad1f9130aa2. Latest consensus state: current_term: 1 leader_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34109 } } }
I20260923 21:02:40.441558 11783 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbdbdd396e9e4ef1a64b7ad1f9130aa2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34109 } } }
I20260923 21:02:40.442528 11783 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [sys.catalog]: This master's current role is: LEADER
I20260923 21:02:40.442895 11784 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2 [sys.catalog]: This master's current role is: LEADER
I20260923 21:02:40.444962 11788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:02:40.450208 11788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:02:40.451452 11777 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 21:02:40.469847 11788 catalog_manager.cc:1347] Generated new cluster ID: bf188455b63d4177a9f00c32454d4d3d
I20260923 21:02:40.470046 11788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:02:40.519570 11788 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:02:40.521728 11788 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:02:40.532526 11788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2: Generated new TSK 0
I20260923 21:02:40.533732 11788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:02:41.338960 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34109
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:02:41 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790197361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 21:02:41.814079 11822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:02:41.814568 11822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:02:41.814744 11822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:02:41.828002 11822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:02:41.828258 11822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:02:41.828508 11822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:02:41.846485 11822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:55701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:02:41.848449 11822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:02:41.850457 11822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:02:41.871958 11832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:02:41.871917 11833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:02:41.880154 11835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:02:41.881254 11822 server_base.cc:1034] running on GCE node
I20260923 21:02:41.882108 11822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:02:41.886430 11822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:02:41.887756 11822 hybrid_clock.cc:648] HybridClock initialized: now 1790197361887688 us; error 64 us; skew 500 ppm
I20260923 21:02:41.888226 11822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:02:41 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:02:41.892864 11822 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:02:41.899202 11822 webserver.cc:460] Webserver started at http://127.0.97.1:36375/ using document root <none> and password file <none>
I20260923 21:02:41.900084 11822 fs_manager.cc:359] Metadata directory not provided
I20260923 21:02:41.900240 11822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:02:41.900646 11822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:02:41 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:02:41.924041 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "0fa8961e162a409298c637ee6ab39805"
format_stamp: "Formatted at 2026-09-23 21:02:41 on dist-test-slave-8xn5"
server_key: "39a3205147244be63d1c85058c10bdc1"
server_key_iv: "575d468a645805522836ceb2e1626b78"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:41.925078 11822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0fa8961e162a409298c637ee6ab39805"
format_stamp: "Formatted at 2026-09-23 21:02:41 on dist-test-slave-8xn5"
server_key: "39a3205147244be63d1c85058c10bdc1"
server_key_iv: "575d468a645805522836ceb2e1626b78"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:41.932255 11822 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260923 21:02:41.968393 11842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:41.970556 11822 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.007s sys 0.000s
I20260923 21:02:41.970781 11822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "0fa8961e162a409298c637ee6ab39805"
format_stamp: "Formatted at 2026-09-23 21:02:41 on dist-test-slave-8xn5"
server_key: "39a3205147244be63d1c85058c10bdc1"
server_key_iv: "575d468a645805522836ceb2e1626b78"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:41.971041 11822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:02:41.996490 11822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:02:42.002938 11822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:02:42.003386 11822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:02:42.005973 11822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:02:42.007862 11822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:02:42.007977 11822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:42.008132 11822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:02:42.008219 11822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:42.107966 11822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44137
I20260923 21:02:42.108016 11954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44137 every 8 connection(s)
I20260923 21:02:42.109791 11822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:02:42.112614 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 11822
I20260923 21:02:42.112811 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:02:42.113538 388 external_mini_cluster.cc:1428] Setting key 13890a7b6d0e61cc1736af2fa63a97eb
Sep 23 21:02:42 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 23 21:02:42 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:02:42.223922 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:42.154052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34109 (local address 127.0.97.1:32969)
0923 21:02:42.172038 (+ 17986us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:42.172113 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:42.179296 (+ 7183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:42.180933 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:42.180948 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:42.181694 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:02:42.194725 (+ 13031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:42.194776 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:42.196547 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:42.196554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:42.198896 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:42.198918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:42.209750 (+ 10832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:42.209760 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:42.210171 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:42.211153 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:02:42.211190 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:02:42.214255 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:02:42.218775 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:02:42.218786 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:02:42.218808 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:02:42.219448 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:02:42.219933 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:02:42.219938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:02:42.219943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:02:42.220232 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:02:42.220965 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:02:42.220973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:02:42.221530 (+ 557us) client_negotiation.cc:769] Sending connection context
0923 21:02:42.221873 (+ 343us) client_negotiation.cc:240] Negotiation successful
0923 21:02:42.222484 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17501,"thread_start_us":19959,"threads_started":1}
I20260923 21:02:42.224208 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:42.153239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32969 (local address 127.0.97.62:34109)
0923 21:02:42.180252 (+ 27013us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:42.180256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:42.180273 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:02:42.180372 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:42.180378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:42.180467 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:42.180597 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:42.195085 (+ 14488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:42.196283 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:42.199098 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:42.199757 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:42.199892 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:42.214647 (+ 14755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:42.214656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:42.214665 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:42.214713 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:42.218575 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:42.219614 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:42.219620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:42.219624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:42.219790 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:42.220380 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:42.220386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:42.220389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:42.220753 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:42.220891 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:42.223529 (+ 2638us) server_negotiation.cc:299] Negotiation successful
0923 21:02:42.223787 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26885,"thread_start_us":26734,"threads_started":1}
I20260923 21:02:42.226586 11955 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34109
I20260923 21:02:42.226927 11955 heartbeater.cc:464] Registering TS with master...
I20260923 21:02:42.227784 11955 heartbeater.cc:511] Master 127.0.97.62:34109 requested a full tablet report, sending...
I20260923 21:02:42.230451 11742 ts_manager.cc:194] Registered new tserver with Master: 0fa8961e162a409298c637ee6ab39805 (127.0.97.1:44137)
I20260923 21:02:42.233003 11742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260923 21:02:42.244693 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.128s user 0.000s sys 0.016s
I20260923 21:02:42.245770 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34109
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:02:42 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 21:02:42.767128 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:02:42.767710 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:02:42.767942 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:02:42.781849 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:02:42.782155 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:02:42.782454 11961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:02:42.800179 11961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:55701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:02:42.802482 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:02:42.804342 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:02:42.826922 11968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:02:42.833987 11969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:02:42.835242 11961 server_base.cc:1034] running on GCE node
W20260923 21:02:42.837124 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:02:42.837934 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:02:42.839272 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:02:42.840698 11961 hybrid_clock.cc:648] HybridClock initialized: now 1790197362840606 us; error 85 us; skew 500 ppm
I20260923 21:02:42.841185 11961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:02:42 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:02:42.845100 11961 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:02:42.859715 11961 webserver.cc:460] Webserver started at http://127.0.97.2:36709/ using document root <none> and password file <none>
I20260923 21:02:42.860580 11961 fs_manager.cc:359] Metadata directory not provided
I20260923 21:02:42.860749 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:02:42.861166 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:02:42 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:02:42.879293 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "c77d2384bee54503b190d5ecdaee7a13"
format_stamp: "Formatted at 2026-09-23 21:02:42 on dist-test-slave-8xn5"
server_key: "32c1a3555c04a9f15dafb8a07c873a2e"
server_key_iv: "0c6a5f115a51e3b6b133b21b44f3f8f9"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:42.880381 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c77d2384bee54503b190d5ecdaee7a13"
format_stamp: "Formatted at 2026-09-23 21:02:42 on dist-test-slave-8xn5"
server_key: "32c1a3555c04a9f15dafb8a07c873a2e"
server_key_iv: "0c6a5f115a51e3b6b133b21b44f3f8f9"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:42.886927 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 21:02:42.905882 11985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:42.907850 11961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:02:42.908041 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "c77d2384bee54503b190d5ecdaee7a13"
format_stamp: "Formatted at 2026-09-23 21:02:42 on dist-test-slave-8xn5"
server_key: "32c1a3555c04a9f15dafb8a07c873a2e"
server_key_iv: "0c6a5f115a51e3b6b133b21b44f3f8f9"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:42.908295 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:02:42.927143 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:02:42.932906 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:02:42.933249 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:02:42.934429 11961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:02:42.936198 11961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:02:42.936290 11961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:42.936450 11961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:02:42.936533 11961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:42.975627 11961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33221
I20260923 21:02:42.975713 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33221 every 8 connection(s)
I20260923 21:02:42.977303 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:02:42.981884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 11961
I20260923 21:02:42.982153 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:02:42.982856 388 external_mini_cluster.cc:1428] Setting key 18eb897f762e83db7785928a56ad1004
Sep 23 21:02:43 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:02:43.022050 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:42.980249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32915 (local address 127.0.97.62:34109)
0923 21:02:43.000383 (+ 20134us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.000390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.000414 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:02:43.002692 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.002701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.002814 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.002941 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:43.004877 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.006150 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.009001 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.009761 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.009907 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.014528 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:43.014538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:43.014546 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:43.014618 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:43.018173 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:43.019226 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:43.019233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:43.019236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:43.019390 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:43.019924 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:43.019929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:43.019932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:43.020281 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:43.020409 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.021423 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.021666 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20015,"thread_start_us":20157,"threads_started":1}
I20260923 21:02:43.023260 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:42.980981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34109 (local address 127.0.97.2:32915)
0923 21:02:42.999773 (+ 18792us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:42.999850 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:43.002317 (+ 2467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:43.003236 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:43.003250 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:43.003846 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:02:43.004681 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.004704 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.006372 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.006379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.008839 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.008857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.010036 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.010043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.010378 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.011232 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:02:43.011268 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:02:43.014175 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:02:43.018422 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:02:43.018432 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:02:43.018453 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:02:43.019073 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:02:43.019541 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:02:43.019546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:02:43.019549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:02:43.019799 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:02:43.020501 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:02:43.020509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:02:43.020952 (+ 443us) client_negotiation.cc:769] Sending connection context
0923 21:02:43.021261 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 21:02:43.021809 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18421,"thread_start_us":18154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:02:43.024963 12098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34109
I20260923 21:02:43.025285 12098 heartbeater.cc:464] Registering TS with master...
I20260923 21:02:43.026120 12098 heartbeater.cc:511] Master 127.0.97.62:34109 requested a full tablet report, sending...
I20260923 21:02:43.031075 11742 ts_manager.cc:194] Registered new tserver with Master: c77d2384bee54503b190d5ecdaee7a13 (127.0.97.2:33221)
I20260923 21:02:43.032203 11742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:02:43.058313 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:55701/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34109
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:02:43.237013 11955 heartbeater.cc:503] Master 127.0.97.62:34109 was elected leader, sending a full tablet report...
W20260923 21:02:43.412887 12104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:02:43.413255 12104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:02:43.413414 12104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:02:43.421162 12104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:02:43.421290 12104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:02:43.421406 12104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:02:43.431588 12104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:55701/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:02:43.433092 12104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:02:43.434448 12104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:02:43.449462 12109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:02:43.449373 12110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:02:43.449373 12112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:02:43.450111 12104 server_base.cc:1034] running on GCE node
I20260923 21:02:43.450940 12104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:02:43.452172 12104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:02:43.453411 12104 hybrid_clock.cc:648] HybridClock initialized: now 1790197363453347 us; error 53 us; skew 500 ppm
I20260923 21:02:43.453867 12104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:02:43 dist-test-slave-8xn5 krb5kdc[11266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:02:43.457283 12104 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:02:43.459463 12104 webserver.cc:460] Webserver started at http://127.0.97.3:35119/ using document root <none> and password file <none>
I20260923 21:02:43.460165 12104 fs_manager.cc:359] Metadata directory not provided
I20260923 21:02:43.460322 12104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:02:43.460654 12104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:02:43 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:02:43.477200 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16"
format_stamp: "Formatted at 2026-09-23 21:02:43 on dist-test-slave-8xn5"
server_key: "1e045a58b0c81b710906657586d25dc8"
server_key_iv: "d34a730afe37096638b5ea8993d44a53"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:43.478008 12104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16"
format_stamp: "Formatted at 2026-09-23 21:02:43 on dist-test-slave-8xn5"
server_key: "1e045a58b0c81b710906657586d25dc8"
server_key_iv: "d34a730afe37096638b5ea8993d44a53"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:43.483436 12104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:02:43.502701 12119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:43.504534 12104 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260923 21:02:43.504763 12104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16"
format_stamp: "Formatted at 2026-09-23 21:02:43 on dist-test-slave-8xn5"
server_key: "1e045a58b0c81b710906657586d25dc8"
server_key_iv: "d34a730afe37096638b5ea8993d44a53"
server_key_version: "kuduclusterkey@0"
I20260923 21:02:43.505023 12104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:02:43.525228 12104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:02:43.529527 12104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:02:43.529847 12104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:02:43.530946 12104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:02:43.532409 12104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:02:43.532514 12104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:43.532667 12104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:02:43.532739 12104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:02:43.570286 12104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42377
I20260923 21:02:43.570320 12231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42377 every 8 connection(s)
I20260923 21:02:43.571599 12104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:02:43.574618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 12104
I20260923 21:02:43.574821 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:02:43.575425 388 external_mini_cluster.cc:1428] Setting key 342e70729ae2315b232c4f5facf877e2
Sep 23 21:02:43 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:02:43.595357 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.574125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45749 (local address 127.0.97.62:34109)
0923 21:02:43.574845 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.574876 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.576402 (+ 1526us) server_negotiation.cc:407] Connection header received
0923 21:02:43.578946 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.578952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.579046 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.579158 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:43.580753 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.581500 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.583451 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.583854 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.583947 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.587914 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:43.587919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:43.587923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:43.587957 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:43.591536 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:43.592340 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:43.592345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:43.592349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:43.592955 (+ 606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:43.593482 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:43.593487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:43.593491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:43.593821 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:43.593977 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.594767 (+ 790us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.594984 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":341,"threads_started":1}
I20260923 21:02:43.596428 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.574721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34109 (local address 127.0.97.3:45749)
0923 21:02:43.575805 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:43.575852 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:43.578589 (+ 2737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:43.579382 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:43.579391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:43.579884 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:02:43.580547 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.580568 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.581762 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.581766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.583312 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.583327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.584090 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.584096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.584317 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.585123 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:02:43.585145 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:02:43.587635 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:02:43.591697 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:02:43.591707 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:02:43.591733 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:02:43.592208 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:02:43.593086 (+ 878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:02:43.593091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:02:43.593095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:02:43.593350 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:02:43.593954 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:02:43.593962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:02:43.594394 (+ 432us) client_negotiation.cc:769] Sending connection context
0923 21:02:43.594679 (+ 285us) client_negotiation.cc:240] Negotiation successful
0923 21:02:43.595251 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":402,"threads_started":1}
I20260923 21:02:43.598043 12232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34109
I20260923 21:02:43.598465 12232 heartbeater.cc:464] Registering TS with master...
I20260923 21:02:43.599202 12232 heartbeater.cc:511] Master 127.0.97.62:34109 requested a full tablet report, sending...
I20260923 21:02:43.600692 11742 ts_manager.cc:194] Registered new tserver with Master: 9f9372be719d4fe0b9cd2dbbf2a33a16 (127.0.97.3:42377)
I20260923 21:02:43.601509 11742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45749
I20260923 21:02:43.604169 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:02:43.622506 12234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.608705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.0.97.62:34109)
0923 21:02:43.608918 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.608926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.609389 (+ 463us) server_negotiation.cc:407] Connection header received
0923 21:02:43.609589 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.609593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.609667 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.609749 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:43.611438 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.612146 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.613580 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.614089 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.614191 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.616743 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:43.616749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:43.616752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:43.616792 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:43.619864 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:43.620543 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:43.620548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:43.620552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:43.620690 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:43.621069 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:43.621074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:43.621076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:43.621394 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:43.621597 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.621950 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.622125 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 21:02:43.627316 11742 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:02:43.635317 11742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:02:43.679039 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.649962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33221 (local address 127.0.0.1:37518)
0923 21:02:43.668434 (+ 18472us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:43.668456 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:43.668569 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:43.669973 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:43.669981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:43.670016 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:02:43.670553 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.670570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.672557 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.672564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.676223 (+ 3659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.676236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.677829 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.677835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.678051 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.678126 (+ 75us) client_negotiation.cc:769] Sending connection context
0923 21:02:43.678297 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 21:02:43.678576 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18349,"mutex_wait_us":114,"thread_start_us":18240,"threads_started":1}
I20260923 21:02:43.679065 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.649975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.0.97.2:33221)
0923 21:02:43.669048 (+ 19073us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.669058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.669084 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:02:43.669196 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.669203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.669447 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.669674 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:02:43.670780 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.672384 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.676425 (+ 4041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.677598 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.677751 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.677963 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.678416 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.678765 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18918,"thread_start_us":18818,"threads_started":1}
I20260923 21:02:43.680080 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.650039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.0.97.3:42377)
0923 21:02:43.668127 (+ 18088us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.668138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.668176 (+ 38us) server_negotiation.cc:407] Connection header received
0923 21:02:43.668284 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.668294 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.668583 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.668892 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:02:43.670013 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.672194 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.676319 (+ 4125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.677625 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.677787 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.678015 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.679529 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.679734 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17901,"thread_start_us":17757,"threads_started":1}
I20260923 21:02:43.679730 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.649338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42377 (local address 127.0.0.1:60376)
0923 21:02:43.666102 (+ 16764us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:43.666164 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:43.666380 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:43.669104 (+ 2724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:43.669111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:43.669153 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:02:43.669718 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.669736 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.672411 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.672420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.676149 (+ 3729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.676165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.678157 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.678164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.678373 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.678408 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:02:43.678485 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 21:02:43.679377 (+ 892us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16403,"mutex_wait_us":779,"thread_start_us":17335,"threads_started":1}
I20260923 21:02:43.682619 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.669262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44137 (local address 127.0.0.1:49816)
0923 21:02:43.674174 (+ 4912us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:43.674207 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:43.674386 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:43.674956 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:43.674960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:43.674989 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:02:43.675471 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.675485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.677216 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.677221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.679481 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.679502 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.681784 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.681791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.681971 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.682014 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 21:02:43.682126 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 21:02:43.682250 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4752,"mutex_wait_us":24,"thread_start_us":4609,"threads_started":1}
I20260923 21:02:43.682956 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.669018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.0.97.1:44137)
0923 21:02:43.673231 (+ 4213us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.673243 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.674315 (+ 1072us) server_negotiation.cc:407] Connection header received
0923 21:02:43.674579 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.674585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.674770 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.674975 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:02:43.675689 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.677049 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.679670 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.681271 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.681416 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.681640 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.682483 (+ 843us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.682652 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3969,"thread_start_us":3774,"threads_started":1}
I20260923 21:02:43.683634 12033 tablet_service.cc:1467] Processing CreateTablet for tablet d11a7265d50e4991945212790c9e1921 (DEFAULT_TABLE table=test-table [id=e9b321d49573438b997dc055fb3d5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:02:43.683809 12167 tablet_service.cc:1467] Processing CreateTablet for tablet d11a7265d50e4991945212790c9e1921 (DEFAULT_TABLE table=test-table [id=e9b321d49573438b997dc055fb3d5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:02:43.685061 12167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d11a7265d50e4991945212790c9e1921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:02:43.685487 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d11a7265d50e4991945212790c9e1921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:02:43.687136 11890 tablet_service.cc:1467] Processing CreateTablet for tablet d11a7265d50e4991945212790c9e1921 (DEFAULT_TABLE table=test-table [id=e9b321d49573438b997dc055fb3d5707]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:02:43.688699 11890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d11a7265d50e4991945212790c9e1921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:02:43.697292 12251 tablet_bootstrap.cc:492] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Bootstrap starting.
I20260923 21:02:43.701622 12251 tablet_bootstrap.cc:654] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Neither blocks nor log segments found. Creating new log.
I20260923 21:02:43.702870 12251 log.cc:824] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Log is configured to *not* fsync() on all Append() calls
I20260923 21:02:43.703109 12252 tablet_bootstrap.cc:492] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Bootstrap starting.
I20260923 21:02:43.704481 12253 tablet_bootstrap.cc:492] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Bootstrap starting.
I20260923 21:02:43.707442 12252 tablet_bootstrap.cc:654] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Neither blocks nor log segments found. Creating new log.
I20260923 21:02:43.708003 12253 tablet_bootstrap.cc:654] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Neither blocks nor log segments found. Creating new log.
I20260923 21:02:43.708271 12251 tablet_bootstrap.cc:492] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: No bootstrap required, opened a new log
I20260923 21:02:43.708562 12251 ts_tablet_manager.cc:1397] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260923 21:02:43.708777 12252 log.cc:824] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Log is configured to *not* fsync() on all Append() calls
I20260923 21:02:43.709156 12253 log.cc:824] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Log is configured to *not* fsync() on all Append() calls
I20260923 21:02:43.712262 12252 tablet_bootstrap.cc:492] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: No bootstrap required, opened a new log
I20260923 21:02:43.712546 12252 ts_tablet_manager.cc:1397] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 21:02:43.712685 12253 tablet_bootstrap.cc:492] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: No bootstrap required, opened a new log
I20260923 21:02:43.712986 12253 ts_tablet_manager.cc:1397] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 21:02:43.720266 12252 raft_consensus.cc:348] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.720674 12252 raft_consensus.cc:374] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:02:43.720788 12252 raft_consensus.cc:729] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77d2384bee54503b190d5ecdaee7a13, State: Initialized, Role: FOLLOWER
I20260923 21:02:43.721439 12252 consensus_queue.cc:260] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.722604 12098 heartbeater.cc:503] Master 127.0.97.62:34109 was elected leader, sending a full tablet report...
I20260923 21:02:43.723166 12252 ts_tablet_manager.cc:1428] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260923 21:02:43.722024 12251 raft_consensus.cc:348] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.723532 12251 raft_consensus.cc:374] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:02:43.723663 12251 raft_consensus.cc:729] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9372be719d4fe0b9cd2dbbf2a33a16, State: Initialized, Role: FOLLOWER
I20260923 21:02:43.723994 12253 raft_consensus.cc:348] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.724411 12253 raft_consensus.cc:374] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:02:43.724321 12251 consensus_queue.cc:260] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.724539 12253 raft_consensus.cc:729] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa8961e162a409298c637ee6ab39805, State: Initialized, Role: FOLLOWER
I20260923 21:02:43.725140 12253 consensus_queue.cc:260] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.727643 12232 heartbeater.cc:503] Master 127.0.97.62:34109 was elected leader, sending a full tablet report...
I20260923 21:02:43.728729 12251 ts_tablet_manager.cc:1428] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Time spent starting tablet: real 0.020s user 0.011s sys 0.007s
I20260923 21:02:43.729401 12253 ts_tablet_manager.cc:1428] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
W20260923 21:02:43.741818 12099 tablet.cc:2369] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:02:43.823289 12233 tablet.cc:2369] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:02:43.889169 12258 raft_consensus.cc:482] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:02:43.889488 12258 raft_consensus.cc:504] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.891223 12258 leader_election.cc:288] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c77d2384bee54503b190d5ecdaee7a13 (127.0.97.2:33221), 0fa8961e162a409298c637ee6ab39805 (127.0.97.1:44137)
I20260923 21:02:43.898223 12235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.891672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33221 (local address 127.0.97.3:46375)
0923 21:02:43.891865 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:43.891890 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:43.892014 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:43.892438 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:43.892442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:43.892471 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:02:43.892754 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.892769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.894309 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.894312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.896215 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.896231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.897686 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.897691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.897817 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.897842 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 21:02:43.897918 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 21:02:43.898012 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":48}
I20260923 21:02:43.898495 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.891731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46375 (local address 127.0.97.2:33221)
0923 21:02:43.891928 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.891932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.891950 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:02:43.892221 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.892234 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.892304 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.892417 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:02:43.892970 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.894172 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.896387 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.897483 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.897619 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.897719 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.897974 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.898169 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 21:02:43.899128 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.892442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42091 (local address 127.0.97.1:44137)
0923 21:02:43.892659 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.892664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.893185 (+ 521us) server_negotiation.cc:407] Connection header received
0923 21:02:43.893289 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.893296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.893375 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.893474 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:02:43.893940 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.894794 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.896964 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.898199 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.898546 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.898706 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.898760 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.898894 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 21:02:43.899621 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.892135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44137 (local address 127.0.97.3:42091)
0923 21:02:43.893075 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0923 21:02:43.893089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 21:02:43.893187 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:02:43.893490 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:02:43.893493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:02:43.893511 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:02:43.893800 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.893808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.894958 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.894965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.896809 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:02:43.896819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.898454 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.898459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:02:43.898608 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.898642 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 21:02:43.898719 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 21:02:43.898857 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":26,"thread_start_us":631,"threads_started":1}
I20260923 21:02:43.900064 12053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11a7265d50e4991945212790c9e1921" candidate_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77d2384bee54503b190d5ecdaee7a13" is_pre_election: true
I20260923 21:02:43.900588 12053 raft_consensus.cc:2454] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9372be719d4fe0b9cd2dbbf2a33a16 in term 0.
I20260923 21:02:43.900681 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11a7265d50e4991945212790c9e1921" candidate_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa8961e162a409298c637ee6ab39805" is_pre_election: true
I20260923 21:02:43.901297 12121 leader_election.cc:302] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9372be719d4fe0b9cd2dbbf2a33a16, c77d2384bee54503b190d5ecdaee7a13; no voters:
I20260923 21:02:43.901342 11910 raft_consensus.cc:2454] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9372be719d4fe0b9cd2dbbf2a33a16 in term 0.
I20260923 21:02:43.901808 12258 raft_consensus.cc:2781] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:02:43.901989 12258 raft_consensus.cc:482] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:02:43.902117 12258 raft_consensus.cc:3037] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:02:43.904326 12258 raft_consensus.cc:504] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.904935 12258 leader_election.cc:288] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [CANDIDATE]: Term 1 election: Requested vote from peers c77d2384bee54503b190d5ecdaee7a13 (127.0.97.2:33221), 0fa8961e162a409298c637ee6ab39805 (127.0.97.1:44137)
I20260923 21:02:43.905578 12053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11a7265d50e4991945212790c9e1921" candidate_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77d2384bee54503b190d5ecdaee7a13"
I20260923 21:02:43.905668 11910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d11a7265d50e4991945212790c9e1921" candidate_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa8961e162a409298c637ee6ab39805"
I20260923 21:02:43.905805 12053 raft_consensus.cc:3037] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:02:43.905862 11910 raft_consensus.cc:3037] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:02:43.909073 11910 raft_consensus.cc:2454] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9372be719d4fe0b9cd2dbbf2a33a16 in term 1.
I20260923 21:02:43.909121 12053 raft_consensus.cc:2454] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9372be719d4fe0b9cd2dbbf2a33a16 in term 1.
I20260923 21:02:43.909518 12121 leader_election.cc:302] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa8961e162a409298c637ee6ab39805, 9f9372be719d4fe0b9cd2dbbf2a33a16; no voters:
I20260923 21:02:43.909854 12258 raft_consensus.cc:2781] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:02:43.910450 12258 raft_consensus.cc:686] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9372be719d4fe0b9cd2dbbf2a33a16, State: Running, Role: LEADER
I20260923 21:02:43.911422 12258 consensus_queue.cc:237] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } }
I20260923 21:02:43.916818 11742 catalog_manager.cc:5495] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f9372be719d4fe0b9cd2dbbf2a33a16 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9372be719d4fe0b9cd2dbbf2a33a16" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 } health_report { overall_health: UNKNOWN } } }
W20260923 21:02:43.917976 11957 tablet.cc:2369] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:02:43.959473 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:43.953954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.0.97.3:42377)
0923 21:02:43.954183 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:43.954188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:43.954203 (+ 15us) server_negotiation.cc:407] Connection header received
0923 21:02:43.954283 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:43.954287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:43.954431 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:43.954595 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:02:43.954967 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.955689 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.957158 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:43.957570 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:43.957709 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:43.958185 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:02:43.958854 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:02:43.959019 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:43.959126 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 21:02:43.959210 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 21:02:43.969023 11910 raft_consensus.cc:1261] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 1 FOLLOWER]: Refusing update from remote peer 9f9372be719d4fe0b9cd2dbbf2a33a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:02:43.968981 12053 raft_consensus.cc:1261] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 1 FOLLOWER]: Refusing update from remote peer 9f9372be719d4fe0b9cd2dbbf2a33a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:02:43.970170 12261 consensus_queue.cc:1035] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa8961e162a409298c637ee6ab39805" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:02:43.970629 12258 consensus_queue.cc:1035] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77d2384bee54503b190d5ecdaee7a13" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:02:43.991797 12265 mvcc.cc:205] Tried to move back new op lower bound from 7332648402803294208 to 7332648402585477120. Current Snapshot: MvccSnapshot[applied={T|T < 7332648402803294208}]
I20260923 21:02:44.009931 11741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:02:44.010265 11741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:02:44.014570 11741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbdbdd396e9e4ef1a64b7ad1f9130aa2: Sending DeleteTablet for 3 replicas of tablet d11a7265d50e4991945212790c9e1921
I20260923 21:02:44.016041 12167 tablet_service.cc:1514] Processing DeleteTablet for tablet d11a7265d50e4991945212790c9e1921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:02:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60376
I20260923 21:02:44.016109 12033 tablet_service.cc:1514] Processing DeleteTablet for tablet d11a7265d50e4991945212790c9e1921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:02:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37518
I20260923 21:02:44.016166 11890 tablet_service.cc:1514] Processing DeleteTablet for tablet d11a7265d50e4991945212790c9e1921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:02:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49816
I20260923 21:02:44.019564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 11822
I20260923 21:02:44.051192 12276 tablet_replica.cc:330] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: stopping tablet replica
I20260923 21:02:44.051841 12276 raft_consensus.cc:2229] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:02:44.052516 12276 raft_consensus.cc:2258] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:02:44.053488 12275 tablet_replica.cc:330] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: stopping tablet replica
I20260923 21:02:44.053988 12275 raft_consensus.cc:2229] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:02:44.054808 12275 raft_consensus.cc:2258] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:02:44.057278 12275 ts_tablet_manager.cc:1905] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:02:44.057458 12274 tablet_replica.cc:330] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: stopping tablet replica
I20260923 21:02:44.057966 12274 raft_consensus.cc:2229] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:02:44.058619 12274 raft_consensus.cc:2258] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:02:44.059782 12276 ts_tablet_manager.cc:1905] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 21:02:44 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:02:44.068470 12275 ts_tablet_manager.cc:1918] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:02:44.068706 12275 log.cc:1196] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/d11a7265d50e4991945212790c9e1921
I20260923 21:02:44.069849 12276 ts_tablet_manager.cc:1918] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:02:44.070057 12276 log.cc:1196] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/d11a7265d50e4991945212790c9e1921
I20260923 21:02:44.070729 12276 ts_tablet_manager.cc:1939] T d11a7265d50e4991945212790c9e1921 P 0fa8961e162a409298c637ee6ab39805: Deleting consensus metadata
I20260923 21:02:44.071239 12274 ts_tablet_manager.cc:1905] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:02:44.072995 11727 catalog_manager.cc:4841] TS 0fa8961e162a409298c637ee6ab39805 (127.0.97.1:44137): tablet d11a7265d50e4991945212790c9e1921 (table test-table [id=e9b321d49573438b997dc055fb3d5707]) successfully deleted
I20260923 21:02:44.075021 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:44.021083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.0.97.1:44137)
0923 21:02:44.021267 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:44.021274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:44.043176 (+ 21902us) server_negotiation.cc:407] Connection header received
0923 21:02:44.043810 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:44.043820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:44.043947 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:44.044066 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:44.046204 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:44.047602 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:44.054368 (+ 6766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:44.055520 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:44.055746 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:44.064081 (+ 8335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:44.064105 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:44.064127 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:44.064188 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:44.069537 (+ 5349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:44.071150 (+ 1613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:44.071162 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:44.071166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:44.071455 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:44.072085 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:44.072094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:44.072098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:44.072835 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:44.073170 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:44.073963 (+ 793us) server_negotiation.cc:299] Negotiation successful
0923 21:02:44.074249 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 21:02:44.076004 11950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:02:44.079849 12274 ts_tablet_manager.cc:1918] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:02:44.080085 12274 log.cc:1196] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/d11a7265d50e4991945212790c9e1921
I20260923 21:02:44.080687 12274 ts_tablet_manager.cc:1939] T d11a7265d50e4991945212790c9e1921 P c77d2384bee54503b190d5ecdaee7a13: Deleting consensus metadata
I20260923 21:02:44.082490 12275 ts_tablet_manager.cc:1939] T d11a7265d50e4991945212790c9e1921 P 9f9372be719d4fe0b9cd2dbbf2a33a16: Deleting consensus metadata
I20260923 21:02:44.082578 11727 catalog_manager.cc:4841] TS c77d2384bee54503b190d5ecdaee7a13 (127.0.97.2:33221): tablet d11a7265d50e4991945212790c9e1921 (table test-table [id=e9b321d49573438b997dc055fb3d5707]) successfully deleted
I20260923 21:02:44.084321 11726 catalog_manager.cc:4841] TS 9f9372be719d4fe0b9cd2dbbf2a33a16 (127.0.97.3:42377): tablet d11a7265d50e4991945212790c9e1921 (table test-table [id=e9b321d49573438b997dc055fb3d5707]) successfully deleted
I20260923 21:02:44.231087 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 11822
I20260923 21:02:44.249066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 11961
Sep 23 21:02:44 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:02:44.268689 12246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:44.249998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.0.97.2:33221)
0923 21:02:44.250221 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:44.250228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:44.250253 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:02:44.250360 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:44.250366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:44.250466 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:44.250591 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:44.252021 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:44.253267 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:44.254804 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:44.255544 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:44.255690 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:44.260170 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:44.260191 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:44.260216 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:44.260266 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:44.264831 (+ 4565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:44.265814 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:44.265826 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:44.265830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:44.266065 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:44.266689 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:44.266696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:44.266700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:44.267293 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:44.267660 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:44.267891 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 21:02:44.268189 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:02:44.269495 12093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:02:44.359444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 11961
I20260923 21:02:44.376893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 12104
Sep 23 21:02:44 dist-test-slave-8xn5 krb5kdc[11266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:02:44.397190 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:02:44.377706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.97.3:42377)
0923 21:02:44.377876 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 21:02:44.377880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:02:44.377976 (+ 96us) server_negotiation.cc:407] Connection header received
0923 21:02:44.378180 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:02:44.378183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:02:44.378244 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:02:44.378425 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:02:44.380133 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:44.380889 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:44.382807 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:02:44.383277 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:02:44.383376 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:02:44.389051 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:02:44.389063 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:02:44.389098 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:02:44.389150 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:02:44.393609 (+ 4459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:44.394509 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:44.394520 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:44.394524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:44.394717 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:02:44.395247 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:02:44.395253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:02:44.395257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:02:44.395724 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:02:44.396045 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 21:02:44.396404 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 21:02:44.396639 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 21:02:44.397987 12227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:02:44.533157 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 12104
I20260923 21:02:44.553050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 11710
I20260923 21:02:44.554287 11772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:02:44.668269 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 11710
I20260923 21:02:44.693140 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 21:02:45.028817 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 21:02:45.029029 388 mini_ranger.cc:66] Stopping Ranger...
I20260923 21:02:45.375427 388 mini_ranger.cc:68] Stopped Ranger
2026-09-23 21:02:45.375 UTC [11296] LOG: received smart shutdown request
2026-09-23 21:02:45.379 UTC [11296] LOG: background worker "logical replication launcher" (PID 11305) exited with exit code 1
2026-09-23 21:02:45.379 UTC [11300] LOG: shutting down
2026-09-23 21:02:45.576 UTC [11296] LOG: database system is shut down
2026-09-23T21:02:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53274 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:02:45 dist-test-slave-8xn5 krb5kdc[12290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:02:45 dist-test-slave-8xn5 krb5kdc[12290](info): set up 2 sockets
Sep 23 21:02:45 dist-test-slave-8xn5 krb5kdc[12290](info): commencing operation
krb5kdc: starting...
W20260923 21:02:46.870750 388 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:02:47 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:02:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260923 21:02:47.221266 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDsUdkt/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/postgres -l logfile start
2026-09-23 21:02:50.004 UTC [12320] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 21:02:50.004 UTC [12320] LOG: listening on IPv4 address "127.0.97.20", port 46707
2026-09-23 21:02:50.009 UTC [12320] LOG: listening on Unix socket "/tmp/.s.PGSQL.46707"
2026-09-23 21:02:50.032 UTC [12323] LOG: database system was shut down at 2026-09-23 21:02:48 UTC
2026-09-23 21:02:50.037 UTC [12320] LOG: database system is ready to accept connections
I20260923 21:02:51.030761 388 mini_postgres.cc:96] Postgres bound to 46707
2026-09-23 21:02:51.048 UTC [12331] FATAL: database "slave" does not exist
127.0.97.20:46707 - accepting connections
I20260923 21:02:51.049144 388 mini_ranger.cc:161] Starting Ranger...
I20260923 21:02:51.077742 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 21:02:51.655836 388 mini_ranger.cc:87] Created ranger Postgres database
I20260923 21:02:51.655972 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-admin
2026-09-23 21:02:51,966 [I] DB FLAVOR :POSTGRES
2026-09-23 21:02:51,967 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 21:02:51,967 [I] Checking connection..
2026-09-23 21:02:51,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 21:02:52,227 [I] Checking connection passed.
2026-09-23 21:02:52,227 [I] --------- Verifying version history table ---------
2026-09-23 21:02:52,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 21:02:52,476 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 21:02:52,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 21:02:52,747 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 21:02:52,747 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 21:02:52,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 21:02:53.012 UTC [12395] WARNING: there is no transaction in progress
2026-09-23 21:02:53,020 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 21:02:53,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 21:02:53,328 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 21:02:53,328 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 21:02:53,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 21:02:53,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 21:02:53,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8xn5.c.gcp-upstream.internal','N') ;"
2026-09-23 21:02:54,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 21:02:54,333 [I] Table x_portal_user does not exist in database ranger
2026-09-23 21:02:54,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 21:02:54,606 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 21:02:54,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 21:02:54,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 21:02:55,104 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 21:02:55,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 21:02:56.202 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.227 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.247 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.264 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.282 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.302 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.363 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.370 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.387 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.398 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.410 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.426 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.438 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.445 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.453 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:56.461 UTC [12503] WARNING: there is no transaction in progress
2026-09-23 21:02:57,217 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 21:02:57,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 21:02:57,480 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 21:02:57,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 21:02:57,741 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 21:02:57,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 21:02:57,986 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 21:02:57,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 21:02:58,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 21:02:58,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8xn5.c.gcp-upstream.internal';"
2026-09-23 21:02:58,698 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 21:02:58,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 21:02:58,945 [I] DB_PATCHES have already been applied
I20260923 21:02:58.951563 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDsUdkt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 21:02:58.951684 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260923 21:02:58.967060 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:60899
Sep 23, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 9:02:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 21:03:00.136054 388 mini_ranger.cc:160] Time spent starting Ranger: real 9.087s user 0.000s sys 0.042s
Sep 23, 2026 9:03:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 9:03:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 23, 2026 9:03:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 9:03:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 9:03:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 21:03:23.582742 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 21:03:23 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 21:03:23.757349 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 21:03:23.779950 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 21:03:24.849859 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 21:03:24.850000 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms
I20260923 21:03:24.850060 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:46707
2026-09-23 21:03:25,249 [I] DB FLAVOR :POSTGRES
2026-09-23 21:03:25,250 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 21:03:25,250 [I] Checking connection
2026-09-23 21:03:25,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 21:03:25,501 [I] connection success
2026-09-23 21:03:25,502 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 21:03:25,502 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 21:03:25,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 21:03:25,770 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 21:03:25,771 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 21:03:25,771 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 21:03:25,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46707/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 21:03:26,100 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 21:03:26.641582 388 mini_ranger_kms.cc:310] Created kms service
I20260923 21:03:26.869786 388 mini_ranger_kms.cc:326] Created kudu user
I20260923 21:03:26.923159 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 21:03:26 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:27.413331 388 mini_ranger_kms.cc:384] Added ranger policy
I20260923 21:03:27.413676 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDsUdkt/build/asan/bin/postgresql.jar:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDsUdkt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 21:03:27.413725 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260923 21:03:27.429181 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12684
I20260923 21:03:27.429329 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:53253
Sep 23, 2026 9:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 9:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 9:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 9:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 9:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 9:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 21:03:29.816793 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.059s user 0.075s sys 0.335s
I20260923 21:03:29.817063 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 21:03:29.817239 388 mini_ranger_kms.cc:401] 127.0.97.20:53253/kms/v1/keys
Sep 23 21:03:29 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790197409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 21:03:30 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 21:03:30 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 21:03:33 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:03:33.370811 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33689
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33689
--ranger_config_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:33.736907 12734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:33.737234 12734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:33.737386 12734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:33.744927 12734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:33.745086 12734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:33.745200 12734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:33.745309 12734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:33.745399 12734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:33.756343 12734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:53253/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33689
--ranger_config_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:33.757771 12734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:33.759305 12734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:33.774063 12742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:33.774284 12739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:33.774264 12740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:33.775388 12734 server_base.cc:1034] running on GCE node
I20260923 21:03:33.776335 12734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:33.778062 12734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:33.779349 12734 hybrid_clock.cc:648] HybridClock initialized: now 1790197413779311 us; error 35 us; skew 500 ppm
I20260923 21:03:33.779774 12734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:33 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:33.783299 12734 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:03:33.785199 12734 webserver.cc:460] Webserver started at http://127.0.97.62:35783/ using document root <none> and password file <none>
I20260923 21:03:33.785928 12734 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:33.786065 12734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:33.786490 12734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:03:33 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:33.853516 12734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "baa5915c3ff44563afcc750419bdc4e1"
format_stamp: "Formatted at 2026-09-23 21:03:33 on dist-test-slave-8xn5"
server_key: "108b160a35d94d7d4255e4a9ed0a3abd"
server_key_iv: "1466a77fef67aec6815f410428d9675f"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:33.854539 12734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "baa5915c3ff44563afcc750419bdc4e1"
format_stamp: "Formatted at 2026-09-23 21:03:33 on dist-test-slave-8xn5"
server_key: "108b160a35d94d7d4255e4a9ed0a3abd"
server_key_iv: "1466a77fef67aec6815f410428d9675f"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:33.861886 12734 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260923 21:03:33.883996 12749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:33.886184 12734 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:03:33.886493 12734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "baa5915c3ff44563afcc750419bdc4e1"
format_stamp: "Formatted at 2026-09-23 21:03:33 on dist-test-slave-8xn5"
server_key: "108b160a35d94d7d4255e4a9ed0a3abd"
server_key_iv: "1466a77fef67aec6815f410428d9675f"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:33.886797 12734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:33.926539 12734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:33.931789 12734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:33.932170 12734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:33.954232 12734 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33689
I20260923 21:03:33.954298 12800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33689 every 8 connection(s)
I20260923 21:03:33.955825 12734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:03:33.957895 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 12734
I20260923 21:03:33.958124 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:03:33.958909 388 external_mini_cluster.cc:1428] Setting key 3aa13c201ff36757687fce83c7201097
I20260923 21:03:33.961229 12801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:33.978256 12801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Bootstrap starting.
I20260923 21:03:33.982208 12801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:33.983700 12801 log.cc:824] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:03:33 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:33.987516 12801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: No bootstrap required, opened a new log
I20260923 21:03:33.997951 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:33.961382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.0.97.62:33689)
0923 21:03:33.971757 (+ 10375us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:33.971775 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:33.971832 (+ 57us) server_negotiation.cc:407] Connection header received
0923 21:03:33.973843 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:33.973886 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:33.974563 (+ 677us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:33.975115 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:33.976779 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:33.978804 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:33.980856 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:33.982044 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:33.982281 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:33.987539 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:33.987553 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:33.987575 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:33.987627 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:33.992146 (+ 4519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:33.993185 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:33.993195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:33.993211 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:33.993426 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:33.994040 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:33.994045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:33.994048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:33.995024 (+ 976us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:33.995376 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:33.995897 (+ 521us) server_negotiation.cc:299] Negotiation successful
0923 21:03:33.996374 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10092,"thread_start_us":9815,"threads_started":1}
I20260923 21:03:33.997704 12801 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:33.998225 12801 raft_consensus.cc:374] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:33.998389 12801 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5915c3ff44563afcc750419bdc4e1, State: Initialized, Role: FOLLOWER
I20260923 21:03:33.999112 12801 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:33.999388 12801 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:03:33.999559 12801 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:03:33.999748 12801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:34.002375 12801 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:34.002943 12801 leader_election.cc:302] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa5915c3ff44563afcc750419bdc4e1; no voters:
I20260923 21:03:34.003621 12801 leader_election.cc:288] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:03:34.003736 12806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:03:34.004356 12806 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 LEADER]: Becoming Leader. State: Replica: baa5915c3ff44563afcc750419bdc4e1, State: Running, Role: LEADER
I20260923 21:03:34.005168 12806 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:34.012159 12808 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baa5915c3ff44563afcc750419bdc4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } }
I20260923 21:03:34.012668 12808 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:34.012844 12807 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa5915c3ff44563afcc750419bdc4e1. Latest consensus state: current_term: 1 leader_uuid: "baa5915c3ff44563afcc750419bdc4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } }
I20260923 21:03:34.012507 12801 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:34.014444 12807 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:34.013659 12810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:03:34.027772 12810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:03:34.052663 12810 catalog_manager.cc:1347] Generated new cluster ID: 60ef15421dc64012ba6c9ef410ec30b8
I20260923 21:03:34.052802 12810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:03:34.067412 12810 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:03:34.069514 12810 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:03:34.095266 12810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Generated new TSK 0
I20260923 21:03:34.096678 12810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:03:34.097714 12801 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:03:35.043941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33689
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:03:35 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790197415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 21:03:35.485213 12846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:35.485535 12846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:35.485685 12846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:35.493868 12846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:35.494058 12846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:35.494282 12846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:03:35.506875 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:53253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:35.508704 12846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:35.510290 12846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:35.531857 12857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:35.538874 12856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:35.539728 12859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:35.541384 12846 server_base.cc:1034] running on GCE node
I20260923 21:03:35.542352 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:35.544173 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:35.545511 12846 hybrid_clock.cc:648] HybridClock initialized: now 1790197415545433 us; error 67 us; skew 500 ppm
I20260923 21:03:35.545967 12846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:35 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:35.550915 12846 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:03:35.554739 12846 webserver.cc:460] Webserver started at http://127.0.97.1:34749/ using document root <none> and password file <none>
I20260923 21:03:35.555833 12846 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:35.556011 12846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:35.556391 12846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:03:35 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:35.583244 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f"
format_stamp: "Formatted at 2026-09-23 21:03:35 on dist-test-slave-8xn5"
server_key: "5f87af1739ae3f56c37c18cfdef757d4"
server_key_iv: "a85f9595e34010b924cc30c17db8a56d"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:35.584501 12846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f"
format_stamp: "Formatted at 2026-09-23 21:03:35 on dist-test-slave-8xn5"
server_key: "5f87af1739ae3f56c37c18cfdef757d4"
server_key_iv: "a85f9595e34010b924cc30c17db8a56d"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:35.592767 12846 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260923 21:03:35.622129 12866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:35.631991 12846 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.005s
I20260923 21:03:35.632357 12846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f"
format_stamp: "Formatted at 2026-09-23 21:03:35 on dist-test-slave-8xn5"
server_key: "5f87af1739ae3f56c37c18cfdef757d4"
server_key_iv: "a85f9595e34010b924cc30c17db8a56d"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:35.632738 12846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:35.661113 12846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:35.665689 12846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:35.666114 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:35.667395 12846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:35.681770 12846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:35.681988 12846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 21:03:35.682224 12846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:35.682382 12846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:35.758714 12846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33713
I20260923 21:03:35.758759 12978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33713 every 8 connection(s)
I20260923 21:03:35.760751 12846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:03:35.769927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 12846
I20260923 21:03:35.770167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:03:35.771175 388 external_mini_cluster.cc:1428] Setting key 75ad853d1384157ce95632e5f4dd7dfe
Sep 23 21:03:35 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:35.796991 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:35.769215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48349 (local address 127.0.97.62:33689)
0923 21:03:35.773732 (+ 4517us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:35.773741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:35.773764 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:03:35.775724 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:35.775731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:35.775904 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:35.776015 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:35.778399 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:35.779625 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:35.782405 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:35.783194 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:35.783302 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:35.788610 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:35.788620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:35.788625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:35.788672 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:35.792720 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:35.793986 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:35.793993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:35.793996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:35.794169 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:35.794766 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:35.794772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:35.794775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:35.795139 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:35.795257 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:35.796326 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0923 21:03:35.796593 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4378,"thread_start_us":12621,"threads_started":1}
I20260923 21:03:35.799974 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:35.768396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.97.1:48349)
0923 21:03:35.769421 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:35.769489 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:35.775106 (+ 5617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:35.776523 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:35.776537 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:35.777223 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:35.778131 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:35.778161 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:35.779883 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:35.779889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:35.782170 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:35.782190 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:35.783712 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:35.783719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:35.784046 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:35.785003 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:35.785037 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:35.788250 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:35.792988 (+ 4738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:35.793001 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:35.793025 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:35.793829 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:35.794350 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:35.794355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:35.794359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:35.794632 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:35.795360 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:35.795368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:35.795849 (+ 481us) client_negotiation.cc:769] Sending connection context
0923 21:03:35.797565 (+ 1716us) client_negotiation.cc:240] Negotiation successful
0923 21:03:35.798196 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":6575,"threads_started":1}
I20260923 21:03:35.802356 12979 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33689
I20260923 21:03:35.802763 12979 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:35.803710 12979 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
I20260923 21:03:35.806427 12766 ts_manager.cc:194] Registered new tserver with Master: 8f3c8be2a82e4fa6a8670dc7b2a5578f (127.0.97.1:33713)
I20260923 21:03:35.808482 12766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48349
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:03:35.861775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33689
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:03:35 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 23 21:03:36 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 21:03:36.405812 12985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:36.406268 12985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:36.406464 12985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:36.419780 12985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:36.419966 12985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:36.420225 12985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:03:36.436542 12985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:53253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:36.438624 12985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:36.440676 12985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:36.471374 12993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:36.474882 12992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:36.484297 12995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:36.485155 12985 server_base.cc:1034] running on GCE node
I20260923 21:03:36.485922 12985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:36.487185 12985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:36.488473 12985 hybrid_clock.cc:648] HybridClock initialized: now 1790197416488397 us; error 66 us; skew 500 ppm
I20260923 21:03:36.488947 12985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:36 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:36.493161 12985 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:03:36.495415 12985 webserver.cc:460] Webserver started at http://127.0.97.2:42771/ using document root <none> and password file <none>
I20260923 21:03:36.496240 12985 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:36.496413 12985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:36.496773 12985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:03:36 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:36.517211 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "5e8eb31cf83044c8bddfb10ad1f2428a"
format_stamp: "Formatted at 2026-09-23 21:03:36 on dist-test-slave-8xn5"
server_key: "90c9049521d80e0a7094d5f2e511da16"
server_key_iv: "1bf1d198f3ce103141281df54a801ec6"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:36.518553 12985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5e8eb31cf83044c8bddfb10ad1f2428a"
format_stamp: "Formatted at 2026-09-23 21:03:36 on dist-test-slave-8xn5"
server_key: "90c9049521d80e0a7094d5f2e511da16"
server_key_iv: "1bf1d198f3ce103141281df54a801ec6"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:36.533723 12985 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.009s sys 0.004s
I20260923 21:03:36.564776 13009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:36.567667 12985 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.007s
I20260923 21:03:36.568118 12985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "5e8eb31cf83044c8bddfb10ad1f2428a"
format_stamp: "Formatted at 2026-09-23 21:03:36 on dist-test-slave-8xn5"
server_key: "90c9049521d80e0a7094d5f2e511da16"
server_key_iv: "1bf1d198f3ce103141281df54a801ec6"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:36.568593 12985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:36.601536 12985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:36.608300 12985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:36.608753 12985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:36.610356 12985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:36.612639 12985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:36.612766 12985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:36.612993 12985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:36.613090 12985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:36.668097 12985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37695
I20260923 21:03:36.668114 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37695 every 8 connection(s)
I20260923 21:03:36.670421 12985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:03:36.671365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 12985
I20260923 21:03:36.671533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:03:36.672094 388 external_mini_cluster.cc:1428] Setting key bae32ebf0bf224205abeffd8cf3bf03c
Sep 23 21:03:36 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:03:36.717674 13123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:36.680741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40017 (local address 127.0.97.62:33689)
0923 21:03:36.694918 (+ 14177us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:36.694926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:36.694965 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:03:36.696433 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:36.696440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:36.696559 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:36.696702 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:36.699244 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:36.700408 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:36.703196 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:36.703830 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:36.703945 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:36.708859 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:36.708868 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:36.708874 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:36.708926 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:36.712576 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:36.713757 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:36.713764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:36.713768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:36.713953 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:36.714818 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:36.714823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:36.714827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:36.715283 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:36.715432 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:36.716738 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0923 21:03:36.717090 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13988,"thread_start_us":13834,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:03:36.718643 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:36.681564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.97.2:40017)
0923 21:03:36.692030 (+ 10466us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:36.692111 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:36.695972 (+ 3861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:36.697220 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:36.697235 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:36.697916 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:36.698965 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:36.699003 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:36.700633 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:36.700639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:36.702974 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:36.702993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:36.704327 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:36.704334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:36.704644 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:36.705547 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:36.705582 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:36.708530 (+ 2948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:36.712849 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:36.712860 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:36.712881 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:36.713576 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:36.714353 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:36.714357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:36.714361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:36.714670 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:36.715536 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:36.715545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:36.716149 (+ 604us) client_negotiation.cc:769] Sending connection context
0923 21:03:36.716496 (+ 347us) client_negotiation.cc:240] Negotiation successful
0923 21:03:36.717142 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10068,"thread_start_us":8670,"threads_started":1}
I20260923 21:03:36.722419 13122 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33689
I20260923 21:03:36.722787 13122 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:36.723708 13122 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
I20260923 21:03:36.725562 12766 ts_manager.cc:194] Registered new tserver with Master: 5e8eb31cf83044c8bddfb10ad1f2428a (127.0.97.2:37695)
I20260923 21:03:36.726688 12766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:03:36.758757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33689
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:36.812885 12979 heartbeater.cc:503] Master 127.0.97.62:33689 was elected leader, sending a full tablet report...
W20260923 21:03:37.204092 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:37.204447 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:37.204628 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:37.212213 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:37.212394 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:37.212742 13128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:03:37.223693 13128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:53253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:37.225090 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:37.226529 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:37.240762 13133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:37.240866 13134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:37.240738 13136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:37.244287 13128 server_base.cc:1034] running on GCE node
I20260923 21:03:37.245170 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:37.246290 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:37.247627 13128 hybrid_clock.cc:648] HybridClock initialized: now 1790197417247525 us; error 87 us; skew 500 ppm
I20260923 21:03:37.248031 13128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:37 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:37.252063 13128 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:03:37.253827 13128 webserver.cc:460] Webserver started at http://127.0.97.3:39633/ using document root <none> and password file <none>
I20260923 21:03:37.254686 13128 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:37.254825 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:37.255183 13128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 21:03:37 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:37.270606 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "857f244397674859b0b95b00878a66ff"
format_stamp: "Formatted at 2026-09-23 21:03:37 on dist-test-slave-8xn5"
server_key: "b861085f2a15328746d3ad361c497a11"
server_key_iv: "fb761cb132d6cfffe9cc784b0c2ca0d2"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:37.271922 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "857f244397674859b0b95b00878a66ff"
format_stamp: "Formatted at 2026-09-23 21:03:37 on dist-test-slave-8xn5"
server_key: "b861085f2a15328746d3ad361c497a11"
server_key_iv: "fb761cb132d6cfffe9cc784b0c2ca0d2"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:37.279773 13128 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260923 21:03:37.298564 13143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:37.300298 13128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:03:37.300547 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "857f244397674859b0b95b00878a66ff"
format_stamp: "Formatted at 2026-09-23 21:03:37 on dist-test-slave-8xn5"
server_key: "b861085f2a15328746d3ad361c497a11"
server_key_iv: "fb761cb132d6cfffe9cc784b0c2ca0d2"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:37.300794 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:37.318310 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:37.323036 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:37.323421 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:37.324481 13128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:37.325842 13128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:37.325917 13128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:37.326004 13128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:37.326049 13128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:37.360253 13128 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40325
I20260923 21:03:37.360289 13255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40325 every 8 connection(s)
I20260923 21:03:37.361735 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:03:37.363252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 13128
I20260923 21:03:37.363454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:03:37.364037 388 external_mini_cluster.cc:1428] Setting key 924b2275003f18ad6cf9871c3663503b
Sep 23 21:03:37 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:37.383818 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:37.364744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45261 (local address 127.0.97.62:33689)
0923 21:03:37.365599 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:37.365606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:37.366520 (+ 914us) server_negotiation.cc:407] Connection header received
0923 21:03:37.368752 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:37.368756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:37.368822 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:37.369022 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:37.370749 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.371520 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.373303 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.373738 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.373842 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:37.377467 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:37.377472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:37.377475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:37.377512 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:37.380804 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.381581 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.381585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.381588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.381731 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.382148 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.382151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.382153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.382535 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:37.382666 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:37.383388 (+ 722us) server_negotiation.cc:299] Negotiation successful
0923 21:03:37.383561 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":449,"threads_started":1}
I20260923 21:03:37.384778 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:37.365213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.97.3:45261)
0923 21:03:37.366281 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:37.366395 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:37.368477 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:37.369373 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:37.369384 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:37.369849 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:37.370554 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:37.370587 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.371755 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.371759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:37.373143 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:37.373162 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.373933 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.373957 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:37.374214 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:37.374913 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:37.374935 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:37.377226 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:37.380942 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:37.380951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:37.380966 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:37.381418 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:37.381831 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:37.381834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:37.381836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:37.382031 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:37.382689 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:37.382700 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:37.383018 (+ 318us) client_negotiation.cc:769] Sending connection context
0923 21:03:37.383261 (+ 243us) client_negotiation.cc:240] Negotiation successful
0923 21:03:37.383669 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":435,"threads_started":1}
I20260923 21:03:37.386282 13256 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33689
I20260923 21:03:37.386617 13256 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:37.387302 13256 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
I20260923 21:03:37.388748 12766 ts_manager.cc:194] Registered new tserver with Master: 857f244397674859b0b95b00878a66ff (127.0.97.3:40325)
I20260923 21:03:37.389545 12766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45261
I20260923 21:03:37.392794 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:03:37.394624 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 12846
Sep 23 21:03:37 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:03:37.411979 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:37.395575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58492 (local address 127.0.97.1:33713)
0923 21:03:37.396185 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:37.396194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:37.396227 (+ 33us) server_negotiation.cc:407] Connection header received
0923 21:03:37.396379 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:37.396387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:37.396650 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:37.396826 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:37.398447 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.399475 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.400992 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.401558 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.401717 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:37.406427 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:37.406442 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:37.406458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:37.406502 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:37.409215 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.409929 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.409937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.409939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.410083 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.410613 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.410617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.410619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.410962 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:37.411232 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:37.411491 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 21:03:37.411700 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":260,"threads_started":1}
I20260923 21:03:37.413383 12974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:37.499837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 12846
I20260923 21:03:37.516228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 12985
Sep 23 21:03:37 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:03:37.534061 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:37.517305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.0.97.2:37695)
0923 21:03:37.517855 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:37.517863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:37.517895 (+ 32us) server_negotiation.cc:407] Connection header received
0923 21:03:37.517956 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:37.517962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:37.518281 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:37.518493 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:37.519869 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.520900 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.522411 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.522988 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.523097 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:37.527671 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:37.527684 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:37.527700 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:37.527746 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:37.530908 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.531598 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.531605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.531608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.531750 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.532401 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.532407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.532410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.532794 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:37.533094 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:37.533336 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 21:03:37.533542 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":296,"threads_started":1}
I20260923 21:03:37.535452 13117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:37.619717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 12985
I20260923 21:03:37.639957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 13128
Sep 23 21:03:37 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:03:37.658119 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:37.641278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.0.97.3:40325)
0923 21:03:37.641983 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:37.641992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:37.642022 (+ 30us) server_negotiation.cc:407] Connection header received
0923 21:03:37.642087 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:37.642110 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:37.642411 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:37.642612 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:37.643965 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.645029 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.646399 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:37.646987 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:37.647131 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:37.651998 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:37.652013 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:37.652029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:37.652072 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:37.655072 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.655864 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.655871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.655874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.656013 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:37.656401 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:37.656406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:37.656409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:37.656785 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:37.657108 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:37.657439 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 21:03:37.657762 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":383,"threads_started":1}
I20260923 21:03:37.659690 13251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:37.744931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 13128
I20260923 21:03:37.761018 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 12734
I20260923 21:03:37.762428 12796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:37.873909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 12734
I20260923 21:03:37.898069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33689
--webserver_interface=127.0.97.62
--webserver_port=35783
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33689
--ranger_config_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:38.243491 13268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:38.243842 13268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:38.244011 13268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:38.251751 13268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:38.251897 13268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:38.251987 13268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:38.252069 13268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:38.252141 13268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:38.262851 13268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33689
--ranger_config_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:38.264490 13268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:38.265981 13268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:38.280117 13274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:38.280224 13273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:38.280164 13276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:38.281045 13268 server_base.cc:1034] running on GCE node
I20260923 21:03:38.281833 13268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:38.283434 13268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:38.284677 13268 hybrid_clock.cc:648] HybridClock initialized: now 1790197418284615 us; error 53 us; skew 500 ppm
I20260923 21:03:38.285037 13268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:38 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:38.288308 13268 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:03:38.289909 13268 webserver.cc:460] Webserver started at http://127.0.97.62:35783/ using document root <none> and password file <none>
I20260923 21:03:38.290702 13268 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:38.290853 13268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 21:03:38 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:38.346009 13268 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 21:03:38.349798 13284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:38.351904 13268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:03:38.352164 13268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "baa5915c3ff44563afcc750419bdc4e1"
format_stamp: "Formatted at 2026-09-23 21:03:33 on dist-test-slave-8xn5"
server_key: "108b160a35d94d7d4255e4a9ed0a3abd"
server_key_iv: "1466a77fef67aec6815f410428d9675f"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:38.353490 13268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:38.378041 13268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:38.383497 13268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:38.383876 13268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:38.406242 13268 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33689
I20260923 21:03:38.406275 13335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33689 every 8 connection(s)
I20260923 21:03:38.408100 13268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:03:38.413383 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 13268
I20260923 21:03:38.414263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33713
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33689
--webserver_port=34749
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:38.415547 13336 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:03:38.419094 13336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Bootstrap starting.
I20260923 21:03:38.446740 13336 log.cc:824] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:38.458665 13336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:03:38.459314 13336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Bootstrap complete.
I20260923 21:03:38.468487 13336 raft_consensus.cc:348] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:38.469484 13336 raft_consensus.cc:729] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5915c3ff44563afcc750419bdc4e1, State: Initialized, Role: FOLLOWER
I20260923 21:03:38.470229 13336 consensus_queue.cc:260] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:38.470573 13336 raft_consensus.cc:388] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:03:38.470731 13336 raft_consensus.cc:482] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:03:38.470943 13336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 1 FOLLOWER]: Advancing to term 2
I20260923 21:03:38.475103 13336 raft_consensus.cc:504] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:38.475659 13336 leader_election.cc:302] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baa5915c3ff44563afcc750419bdc4e1; no voters:
I20260923 21:03:38.476328 13336 leader_election.cc:288] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 21:03:38.476473 13340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 21:03:38.479166 13336 sys_catalog.cc:564] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:38.483697 13340 raft_consensus.cc:686] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [term 2 LEADER]: Becoming Leader. State: Replica: baa5915c3ff44563afcc750419bdc4e1, State: Running, Role: LEADER
I20260923 21:03:38.484457 13340 consensus_queue.cc:237] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } }
I20260923 21:03:38.487447 13341 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "baa5915c3ff44563afcc750419bdc4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } }
I20260923 21:03:38.487952 13341 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:38.488612 13342 sys_catalog.cc:455] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baa5915c3ff44563afcc750419bdc4e1. Latest consensus state: current_term: 2 leader_uuid: "baa5915c3ff44563afcc750419bdc4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baa5915c3ff44563afcc750419bdc4e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33689 } } }
I20260923 21:03:38.488950 13342 sys_catalog.cc:458] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:38.492925 13351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:03:38.499620 13351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:03:38.500713 13336 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 21:03:38.501246 13351 catalog_manager.cc:1259] Loaded cluster ID: 60ef15421dc64012ba6c9ef410ec30b8
I20260923 21:03:38.501381 13351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:03:38.505771 13351 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:03:38.507408 13351 catalog_manager.cc:5879] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Loaded TSK: 0
I20260923 21:03:38.509070 13351 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 21:03:38.869940 13338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:38.870445 13338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:38.870631 13338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:38.883101 13338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:38.883253 13338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:38.883467 13338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:03:38.900384 13338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34749
--tserver_master_addrs=127.0.97.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:38.902215 13338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:38.904004 13338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:38.924788 13374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:38.924767 13375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:38.932868 13377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:38.934378 13338 server_base.cc:1034] running on GCE node
I20260923 21:03:38.935271 13338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:38.938671 13338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:38.940052 13338 hybrid_clock.cc:648] HybridClock initialized: now 1790197418939990 us; error 73 us; skew 500 ppm
I20260923 21:03:38.940490 13338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:38 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:38.945411 13338 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:03:38.962823 13338 webserver.cc:460] Webserver started at http://127.0.97.1:34749/ using document root <none> and password file <none>
I20260923 21:03:38.964043 13338 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:38.964251 13338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 21:03:38 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:39.026429 13338 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.008s sys 0.001s
I20260923 21:03:39.030613 13385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:39.032709 13338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:03:39.032938 13338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f"
format_stamp: "Formatted at 2026-09-23 21:03:35 on dist-test-slave-8xn5"
server_key: "5f87af1739ae3f56c37c18cfdef757d4"
server_key_iv: "a85f9595e34010b924cc30c17db8a56d"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:39.034473 13338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:39.104818 13338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:39.110406 13338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:39.110721 13338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:39.112987 13338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:39.121197 13338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:39.121337 13338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:39.121469 13338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:39.121544 13338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:39.186136 13338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33713
I20260923 21:03:39.186311 13498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33713 every 8 connection(s)
I20260923 21:03:39.188166 13338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:03:39.190408 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 13338
I20260923 21:03:39.191171 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:37695
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33689
--webserver_port=42771
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:03:39 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:39.242188 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:39.192443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50009 (local address 127.0.97.62:33689)
0923 21:03:39.210252 (+ 17809us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:39.210279 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:39.214461 (+ 4182us) server_negotiation.cc:407] Connection header received
0923 21:03:39.215007 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:39.215032 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:39.215616 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:39.216101 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:39.218274 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.220735 (+ 2461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.223438 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.224676 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.224924 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:39.229763 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:39.229779 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:39.229807 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:39.229885 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:39.234854 (+ 4969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:39.236037 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:39.236047 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:39.236056 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:39.236302 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:39.236941 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:39.236946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:39.236950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:39.237631 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:39.237821 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:39.240237 (+ 2416us) server_negotiation.cc:299] Negotiation successful
0923 21:03:39.240688 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17385,"thread_start_us":21539,"threads_started":1}
I20260923 21:03:39.243363 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:39.192454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.97.1:50009)
0923 21:03:39.211021 (+ 18567us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:39.211087 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:39.213690 (+ 2603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:39.216321 (+ 2631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:39.216336 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:39.217068 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:39.217996 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:39.218027 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.220975 (+ 2948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.220981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:39.223223 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:39.223254 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.225020 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.225030 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:39.225335 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:39.226316 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:39.226390 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:39.229355 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:39.235116 (+ 5761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:39.235128 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:39.235148 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:39.235832 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:39.236464 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:39.236469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:39.236473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:39.236788 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:39.237922 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:39.237931 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:39.238396 (+ 465us) client_negotiation.cc:769] Sending connection context
0923 21:03:39.238705 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 21:03:39.239309 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18199,"thread_start_us":17954,"threads_started":1}
I20260923 21:03:39.247628 13499 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33689
I20260923 21:03:39.247977 13499 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:39.248843 13499 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
W20260923 21:03:39.252848 13499 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33689 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 21:03:39.253118 13499 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:39.253933 13499 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
I20260923 21:03:39.255380 13499 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:39.255735 13499 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
W20260923 21:03:39.257236 13499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 21:03:39.637368 13503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:39.637878 13503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:39.638092 13503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:39.651436 13503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:39.651633 13503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:39.651858 13503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:03:39.666574 13503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:37695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42771
--tserver_master_addrs=127.0.97.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:39.668771 13503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:39.671054 13503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:39.700016 13520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:39.706234 13517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:39.707096 13503 server_base.cc:1034] running on GCE node
W20260923 21:03:39.707681 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:39.708755 13503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:39.715180 13503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:39.718506 13503 hybrid_clock.cc:648] HybridClock initialized: now 1790197419718358 us; error 194 us; skew 500 ppm
I20260923 21:03:39.719238 13503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:39 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:39.724769 13503 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:03:39.731046 13503 webserver.cc:460] Webserver started at http://127.0.97.2:42771/ using document root <none> and password file <none>
I20260923 21:03:39.732165 13503 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:39.732398 13503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 21:03:39 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:39.776659 13503 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.015s sys 0.004s
I20260923 21:03:39.783190 13530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:39.785673 13503 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.003s
I20260923 21:03:39.785894 13503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "5e8eb31cf83044c8bddfb10ad1f2428a"
format_stamp: "Formatted at 2026-09-23 21:03:36 on dist-test-slave-8xn5"
server_key: "90c9049521d80e0a7094d5f2e511da16"
server_key_iv: "1bf1d198f3ce103141281df54a801ec6"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:39.787606 13503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 23 21:03:39 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790197419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:39.826350 13503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:39.832769 13503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:39.833140 13503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:39.837976 13503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:39.840121 13503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:39.840242 13503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 21:03:39.840390 13503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:39.840478 13503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:39.912617 13503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37695
I20260923 21:03:39.912690 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37695 every 8 connection(s)
I20260923 21:03:39.914809 13503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:03:39.924031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 13503
I20260923 21:03:39.924858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40325
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33689
--webserver_port=39633
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:03:39 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:39.960908 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:39.918613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33803 (local address 127.0.97.62:33689)
0923 21:03:39.938436 (+ 19823us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:39.938445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:39.938470 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:03:39.938541 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:39.938550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:39.938654 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:39.938780 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:39.941455 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.942728 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.945765 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.946543 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.946675 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:39.952350 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:39.952359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:39.952367 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:39.952422 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:39.956714 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:39.957866 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:39.957874 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:39.957878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:39.958058 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:39.958684 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:39.958690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:39.958693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:39.959078 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:39.959202 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:39.960259 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0923 21:03:39.960513 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19615,"thread_start_us":19497,"threads_started":1}
I20260923 21:03:39.963589 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:39.919219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.97.2:33803)
0923 21:03:39.927825 (+ 8606us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:39.927881 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:39.937477 (+ 9596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:39.939354 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:39.939385 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:39.940168 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:39.941167 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:39.941204 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.942973 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.942981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:39.945535 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:39.945566 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:39.947094 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:39.947105 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:39.947503 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:39.948443 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:39.948484 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:39.951892 (+ 3408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:39.956961 (+ 5069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:39.956972 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:39.957008 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:39.957687 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:39.958215 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:39.958221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:39.958225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:39.958548 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:39.959308 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:39.959317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:39.959790 (+ 473us) client_negotiation.cc:769] Sending connection context
0923 21:03:39.961471 (+ 1681us) client_negotiation.cc:240] Negotiation successful
0923 21:03:39.962105 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8288,"thread_start_us":8056,"threads_started":1}
I20260923 21:03:39.965761 13644 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33689
I20260923 21:03:39.966100 13644 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:39.967113 13644 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
I20260923 21:03:39.969282 13301 ts_manager.cc:194] Registered new tserver with Master: 5e8eb31cf83044c8bddfb10ad1f2428a (127.0.97.2:37695)
I20260923 21:03:39.971566 13301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33803
I20260923 21:03:40.257424 13499 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:40.257964 13499 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
I20260923 21:03:40.259781 13301 ts_manager.cc:194] Registered new tserver with Master: 8f3c8be2a82e4fa6a8670dc7b2a5578f (127.0.97.1:33713)
I20260923 21:03:40.261152 13301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50009
Sep 23 21:03:40 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260923 21:03:40.411988 13648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:40.412539 13648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:40.412750 13648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:40.424917 13648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:40.425194 13648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:40.425446 13648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:03:40.441694 13648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:53253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39633
--tserver_master_addrs=127.0.97.62:33689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:40.443925 13648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:40.445881 13648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:40.494635 13654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:40.498807 13653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:40.502606 13648 server_base.cc:1034] running on GCE node
W20260923 21:03:40.503666 13656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:40.504720 13648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:40.514737 13648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:40.518224 13648 hybrid_clock.cc:648] HybridClock initialized: now 1790197420518052 us; error 165 us; skew 500 ppm
I20260923 21:03:40.518757 13648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:40 dist-test-slave-8xn5 krb5kdc[12290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:40.523407 13648 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:03:40.553521 13648 webserver.cc:460] Webserver started at http://127.0.97.3:39633/ using document root <none> and password file <none>
I20260923 21:03:40.554713 13648 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:40.554905 13648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 21:03:40 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:40.594515 13648 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.001s
I20260923 21:03:40.604621 13664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:40.606779 13648 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.007s
I20260923 21:03:40.607000 13648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "857f244397674859b0b95b00878a66ff"
format_stamp: "Formatted at 2026-09-23 21:03:37 on dist-test-slave-8xn5"
server_key: "b861085f2a15328746d3ad361c497a11"
server_key_iv: "fb761cb132d6cfffe9cc784b0c2ca0d2"
server_key_version: "kuduclusterkey@0"
I20260923 21:03:40.608456 13648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:40.648298 13648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:40.653877 13648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:40.654219 13648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:40.661706 13648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:40.670014 13648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:40.670156 13648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:40.670262 13648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:40.670372 13648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 21:03:40 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260923 21:03:40.976528 13644 heartbeater.cc:503] Master 127.0.97.62:33689 was elected leader, sending a full tablet report...
I20260923 21:03:41.153548 13648 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40325
I20260923 21:03:41.153586 13784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40325 every 8 connection(s)
I20260923 21:03:41.155488 13648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:03:41.163437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 13648
Sep 23 21:03:41 dist-test-slave-8xn5 krb5kdc[12290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:41.185395 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.158589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53127 (local address 127.0.97.62:33689)
0923 21:03:41.159170 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.159175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.160293 (+ 1118us) server_negotiation.cc:407] Connection header received
0923 21:03:41.162235 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.162240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.162385 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.162514 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:41.164252 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.165482 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.172753 (+ 7271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.173209 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.173349 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.177288 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:41.177295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:41.177303 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:41.177365 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:41.180980 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.182104 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.182109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.182113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.182311 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.183177 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.183185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.183191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.183631 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:41.183800 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.184712 (+ 912us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.184943 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":364,"threads_started":1}
I20260923 21:03:41.188040 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.164688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.0.97.62:33689)
0923 21:03:41.170847 (+ 6159us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.170854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.171409 (+ 555us) server_negotiation.cc:407] Connection header received
0923 21:03:41.171618 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.171624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.171703 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.171833 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:41.173814 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.175095 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.177024 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.177606 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.177753 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.180997 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:41.181003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:41.181007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:41.181066 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:41.184306 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.185708 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.185713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.185716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.185861 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.186316 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.186321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.186350 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.186695 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:41.186971 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.187382 (+ 411us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.187632 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6008,"thread_start_us":6521,"threads_started":1}
I20260923 21:03:41.189540 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.159299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33689 (local address 127.0.97.3:53127)
0923 21:03:41.160118 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.160158 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:41.161895 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:41.162789 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:41.162799 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:41.163297 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:41.164008 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.164052 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.170451 (+ 6399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.170462 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.172574 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.172591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.173497 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.173506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.173796 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.174722 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:41.174758 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:41.177025 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:41.181144 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:41.181155 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:41.181179 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:41.181858 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:41.182526 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:41.182531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:41.182534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:41.182815 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:41.183792 (+ 977us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:41.183801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:41.184263 (+ 462us) client_negotiation.cc:769] Sending connection context
0923 21:03:41.184589 (+ 326us) client_negotiation.cc:240] Negotiation successful
0923 21:03:41.185138 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":313,"threads_started":1}
I20260923 21:03:41.191761 13785 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33689
I20260923 21:03:41.192142 13785 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:41.192993 13785 heartbeater.cc:511] Master 127.0.97.62:33689 requested a full tablet report, sending...
I20260923 21:03:41.194742 13301 ts_manager.cc:194] Registered new tserver with Master: 857f244397674859b0b95b00878a66ff (127.0.97.3:40325)
I20260923 21:03:41.195688 13301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53127
I20260923 21:03:41.203915 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:03:41.222146 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.207428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.97.62:33689)
0923 21:03:41.207683 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.207693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.208113 (+ 420us) server_negotiation.cc:407] Connection header received
0923 21:03:41.208303 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.208306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.208378 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.208463 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:41.210481 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.211166 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.212990 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.213383 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.213498 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.216758 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:41.216763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:41.216766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:41.216804 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:41.219526 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.220360 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.220364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.220366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.220514 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.220893 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.220896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.220898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.221148 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:41.221331 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.221681 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.221861 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 21:03:41.227592 13301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:03:41.235620 13301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:03:41.265262 13499 heartbeater.cc:503] Master 127.0.97.62:33689 was elected leader, sending a full tablet report...
I20260923 21:03:41.294173 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.251999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40325 (local address 127.0.0.1:50072)
0923 21:03:41.266667 (+ 14668us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.266731 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:41.267042 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:41.285281 (+ 18239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:41.285289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:41.285318 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:41.285897 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.285922 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.288784 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.288788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.291078 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.291106 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.293152 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.293159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.293384 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.293477 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 21:03:41.293593 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 21:03:41.293798 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13676,"thread_start_us":13739,"threads_started":1}
I20260923 21:03:41.295929 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.252822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.0.97.3:40325)
0923 21:03:41.284550 (+ 31728us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.284559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.284598 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:03:41.284714 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.284723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.285034 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.285279 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:41.286213 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.288432 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.291344 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.292881 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.294948 (+ 2067us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.295302 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.295383 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.295563 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31533,"thread_start_us":40844,"threads_started":1}
I20260923 21:03:41.297338 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.252644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58508)
0923 21:03:41.286137 (+ 33493us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.286158 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:41.286301 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:41.287326 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:41.287332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:41.287361 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:41.287698 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.287711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.291180 (+ 3469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.291186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.293559 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.293575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.296363 (+ 2788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.296371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.296587 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.296623 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 21:03:41.296723 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 21:03:41.296899 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33365,"mutex_wait_us":3,"thread_start_us":35896,"threads_started":1}
I20260923 21:03:41.298239 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.253326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.0.97.1:33713)
0923 21:03:41.282241 (+ 28915us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.282252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.286457 (+ 4205us) server_negotiation.cc:407] Connection header received
0923 21:03:41.286561 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.286569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.286838 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.287037 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:41.287951 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.289634 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.294036 (+ 4402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.295760 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.295897 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.296121 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.297672 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.297875 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28700,"thread_start_us":28950,"threads_started":1}
I20260923 21:03:41.300643 13713 tablet_service.cc:1467] Processing CreateTablet for tablet 819a5338b5004741a6efb96e39174367 (DEFAULT_TABLE table=test-table [id=d2330fc8565a49deb6328ee27d42a634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:41.301565 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.252100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37695 (local address 127.0.0.1:41060)
0923 21:03:41.280741 (+ 28641us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.280774 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:41.280969 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:41.290925 (+ 9956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:41.290930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:41.290961 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:41.291518 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.291530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.294841 (+ 3311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.294849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.296977 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.296993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.300699 (+ 3706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.300707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.300932 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.300984 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 21:03:41.301058 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 21:03:41.301197 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28426,"thread_start_us":28997,"threads_started":1}
I20260923 21:03:41.302822 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.252637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.97.2:37695)
0923 21:03:41.290246 (+ 37609us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.290256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.290295 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:03:41.290430 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.290439 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.290697 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.290909 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:41.291725 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.294691 (+ 2966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.297631 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.300412 (+ 2781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.301963 (+ 1551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.302202 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.302278 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.302530 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37422,"thread_start_us":37251,"threads_started":1}
I20260923 21:03:41.302899 13713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819a5338b5004741a6efb96e39174367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:41.302802 13434 tablet_service.cc:1467] Processing CreateTablet for tablet 819a5338b5004741a6efb96e39174367 (DEFAULT_TABLE table=test-table [id=d2330fc8565a49deb6328ee27d42a634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:41.304397 13434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819a5338b5004741a6efb96e39174367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:41.307122 13578 tablet_service.cc:1467] Processing CreateTablet for tablet 819a5338b5004741a6efb96e39174367 (DEFAULT_TABLE table=test-table [id=d2330fc8565a49deb6328ee27d42a634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:41.308652 13578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819a5338b5004741a6efb96e39174367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:41.328229 13807 tablet_bootstrap.cc:492] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: Bootstrap starting.
I20260923 21:03:41.328238 13806 tablet_bootstrap.cc:492] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Bootstrap starting.
I20260923 21:03:41.331089 13808 tablet_bootstrap.cc:492] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Bootstrap starting.
I20260923 21:03:41.332630 13806 tablet_bootstrap.cc:654] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:41.334282 13806 log.cc:824] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:41.334465 13807 tablet_bootstrap.cc:654] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:41.334605 13808 tablet_bootstrap.cc:654] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:41.335863 13808 log.cc:824] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:41.336367 13807 log.cc:824] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:41.345096 13806 tablet_bootstrap.cc:492] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: No bootstrap required, opened a new log
I20260923 21:03:41.345566 13806 ts_tablet_manager.cc:1397] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20260923 21:03:41.346395 13807 tablet_bootstrap.cc:492] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: No bootstrap required, opened a new log
I20260923 21:03:41.346724 13807 ts_tablet_manager.cc:1397] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260923 21:03:41.357914 13808 tablet_bootstrap.cc:492] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: No bootstrap required, opened a new log
I20260923 21:03:41.358394 13808 ts_tablet_manager.cc:1397] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.014s
I20260923 21:03:41.359032 13806 raft_consensus.cc:348] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.359048 13807 raft_consensus.cc:348] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.359611 13806 raft_consensus.cc:374] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:41.359660 13807 raft_consensus.cc:374] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:41.359736 13806 raft_consensus.cc:729] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857f244397674859b0b95b00878a66ff, State: Initialized, Role: FOLLOWER
I20260923 21:03:41.359856 13807 raft_consensus.cc:729] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3c8be2a82e4fa6a8670dc7b2a5578f, State: Initialized, Role: FOLLOWER
I20260923 21:03:41.360536 13806 consensus_queue.cc:260] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.360762 13807 consensus_queue.cc:260] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.369400 13807 ts_tablet_manager.cc:1428] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: Time spent starting tablet: real 0.023s user 0.013s sys 0.007s
I20260923 21:03:41.372519 13808 raft_consensus.cc:348] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.373974 13785 heartbeater.cc:503] Master 127.0.97.62:33689 was elected leader, sending a full tablet report...
I20260923 21:03:41.374925 13806 ts_tablet_manager.cc:1428] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Time spent starting tablet: real 0.029s user 0.017s sys 0.009s
I20260923 21:03:41.375955 13808 raft_consensus.cc:374] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:41.376137 13808 raft_consensus.cc:729] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8eb31cf83044c8bddfb10ad1f2428a, State: Initialized, Role: FOLLOWER
I20260923 21:03:41.377019 13808 consensus_queue.cc:260] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.380213 13808 ts_tablet_manager.cc:1428] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Time spent starting tablet: real 0.022s user 0.008s sys 0.009s
W20260923 21:03:41.407471 13786 tablet.cc:2369] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:41.426097 13812 raft_consensus.cc:482] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:03:41.426474 13812 raft_consensus.cc:504] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.428508 13812 leader_election.cc:288] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 857f244397674859b0b95b00878a66ff (127.0.97.3:40325), 5e8eb31cf83044c8bddfb10ad1f2428a (127.0.97.2:37695)
W20260923 21:03:41.428582 13645 tablet.cc:2369] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:41.438489 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.429242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44261 (local address 127.0.97.3:40325)
0923 21:03:41.429475 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.429482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.430074 (+ 592us) server_negotiation.cc:407] Connection header received
0923 21:03:41.430460 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.430471 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.430563 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.430641 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:41.431589 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.432813 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.435872 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.437301 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.437459 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.437578 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.437984 (+ 406us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.438127 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 21:03:41.438695 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.430707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60087 (local address 127.0.97.2:37695)
0923 21:03:41.430917 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.430924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.431459 (+ 535us) server_negotiation.cc:407] Connection header received
0923 21:03:41.431725 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.431730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.431804 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.431934 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:41.432460 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.433470 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.436244 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.437597 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.437785 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.437912 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.438224 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.438393 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 21:03:41.438884 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.429868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37695 (local address 127.0.97.1:60087)
0923 21:03:41.431308 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.431330 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:41.431491 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:41.431949 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:41.431954 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:41.431979 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:41.432299 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.432308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.433692 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.433698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.436052 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.436068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.437861 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.437869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.438050 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.438077 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 21:03:41.438197 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 21:03:41.438392 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1288,"mutex_wait_us":60,"thread_start_us":1051,"threads_started":1}
I20260923 21:03:41.439738 13815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.428865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40325 (local address 127.0.97.1:44261)
0923 21:03:41.429946 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.429972 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:41.430220 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:41.430909 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:41.430917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:41.430954 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:41.431430 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.431445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.432970 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.432976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.435660 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:41.435679 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.437546 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.437552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:41.437759 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.437800 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 21:03:41.439171 (+ 1371us) client_negotiation.cc:240] Negotiation successful
0923 21:03:41.439332 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":55,"thread_start_us":663,"threads_started":1}
I20260923 21:03:41.440047 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819a5338b5004741a6efb96e39174367" candidate_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" is_pre_election: true
I20260923 21:03:41.440727 13598 raft_consensus.cc:2454] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3c8be2a82e4fa6a8670dc7b2a5578f in term 0.
I20260923 21:03:41.440901 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819a5338b5004741a6efb96e39174367" candidate_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857f244397674859b0b95b00878a66ff" is_pre_election: true
W20260923 21:03:41.441408 13500 tablet.cc:2369] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:41.441543 13735 raft_consensus.cc:2454] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3c8be2a82e4fa6a8670dc7b2a5578f in term 0.
I20260923 21:03:41.442410 13387 leader_election.cc:302] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8eb31cf83044c8bddfb10ad1f2428a, 8f3c8be2a82e4fa6a8670dc7b2a5578f; no voters:
I20260923 21:03:41.442924 13812 raft_consensus.cc:2781] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:03:41.443096 13812 raft_consensus.cc:482] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:03:41.443208 13812 raft_consensus.cc:3037] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:41.445657 13812 raft_consensus.cc:504] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.446482 13812 leader_election.cc:288] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [CANDIDATE]: Term 1 election: Requested vote from peers 857f244397674859b0b95b00878a66ff (127.0.97.3:40325), 5e8eb31cf83044c8bddfb10ad1f2428a (127.0.97.2:37695)
I20260923 21:03:41.447059 13735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819a5338b5004741a6efb96e39174367" candidate_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857f244397674859b0b95b00878a66ff"
I20260923 21:03:41.447077 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819a5338b5004741a6efb96e39174367" candidate_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a"
I20260923 21:03:41.447332 13735 raft_consensus.cc:3037] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:41.447332 13598 raft_consensus.cc:3037] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:41.451038 13735 raft_consensus.cc:2454] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3c8be2a82e4fa6a8670dc7b2a5578f in term 1.
I20260923 21:03:41.451038 13598 raft_consensus.cc:2454] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3c8be2a82e4fa6a8670dc7b2a5578f in term 1.
I20260923 21:03:41.451673 13387 leader_election.cc:302] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e8eb31cf83044c8bddfb10ad1f2428a, 8f3c8be2a82e4fa6a8670dc7b2a5578f; no voters:
I20260923 21:03:41.452041 13812 raft_consensus.cc:2781] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:03:41.452666 13812 raft_consensus.cc:686] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 1 LEADER]: Becoming Leader. State: Replica: 8f3c8be2a82e4fa6a8670dc7b2a5578f, State: Running, Role: LEADER
I20260923 21:03:41.453286 13812 consensus_queue.cc:237] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } }
I20260923 21:03:41.458948 13301 catalog_manager.cc:5495] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f3c8be2a82e4fa6a8670dc7b2a5578f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f3c8be2a82e4fa6a8670dc7b2a5578f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33713 } health_report { overall_health: HEALTHY } } }
I20260923 21:03:41.490195 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.483208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.97.1:33713)
0923 21:03:41.483622 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.483629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.483654 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:03:41.483813 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.483817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.483914 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.484089 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:03:41.484536 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.485440 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.487385 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.487916 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.488115 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.488683 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:03:41.489282 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:03:41.489485 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.489615 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.489729 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260923 21:03:41.501104 13598 raft_consensus.cc:1261] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 1 FOLLOWER]: Refusing update from remote peer 8f3c8be2a82e4fa6a8670dc7b2a5578f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:41.501104 13735 raft_consensus.cc:1261] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 1 FOLLOWER]: Refusing update from remote peer 8f3c8be2a82e4fa6a8670dc7b2a5578f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:41.502491 13812 consensus_queue.cc:1035] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8eb31cf83044c8bddfb10ad1f2428a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:41.503386 13812 consensus_queue.cc:1035] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [LEADER]: Connected to new peer: Peer: permanent_uuid: "857f244397674859b0b95b00878a66ff" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:41.536157 13820 mvcc.cc:205] Tried to move back new op lower bound from 7332648638455533568 to 7332648638277206016. Current Snapshot: MvccSnapshot[applied={T|T < 7332648638455533568}]
I20260923 21:03:41.540148 13821 mvcc.cc:205] Tried to move back new op lower bound from 7332648638455533568 to 7332648638277206016. Current Snapshot: MvccSnapshot[applied={T|T < 7332648638455533568}]
I20260923 21:03:41.566820 13822 mvcc.cc:205] Tried to move back new op lower bound from 7332648638455533568 to 7332648638277206016. Current Snapshot: MvccSnapshot[applied={T|T < 7332648638455533568}]
I20260923 21:03:41.576789 13300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:03:41.577198 13300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:03:41.582515 13300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P baa5915c3ff44563afcc750419bdc4e1: Sending DeleteTablet for 3 replicas of tablet 819a5338b5004741a6efb96e39174367
I20260923 21:03:41.584329 13713 tablet_service.cc:1514] Processing DeleteTablet for tablet 819a5338b5004741a6efb96e39174367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50072
I20260923 21:03:41.585109 13578 tablet_service.cc:1514] Processing DeleteTablet for tablet 819a5338b5004741a6efb96e39174367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41060
I20260923 21:03:41.585637 13434 tablet_service.cc:1514] Processing DeleteTablet for tablet 819a5338b5004741a6efb96e39174367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58508
I20260923 21:03:41.595366 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 13338
I20260923 21:03:41.595865 13829 tablet_replica.cc:330] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: stopping tablet replica
I20260923 21:03:41.599082 13829 raft_consensus.cc:2229] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:03:41.599988 13829 raft_consensus.cc:2258] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:41.604928 13829 ts_tablet_manager.cc:1905] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:41.610584 13830 tablet_replica.cc:330] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: stopping tablet replica
I20260923 21:03:41.611384 13830 raft_consensus.cc:2229] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:03:41.612062 13830 raft_consensus.cc:2258] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:41.618085 13830 ts_tablet_manager.cc:1905] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:41.625512 13831 tablet_replica.cc:330] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: stopping tablet replica
I20260923 21:03:41.626441 13831 raft_consensus.cc:2229] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:03:41.632062 13831 raft_consensus.cc:2258] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:41.634644 13830 ts_tablet_manager.cc:1918] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:41.634931 13830 log.cc:1196] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/819a5338b5004741a6efb96e39174367
I20260923 21:03:41.635685 13830 ts_tablet_manager.cc:1939] T 819a5338b5004741a6efb96e39174367 P 5e8eb31cf83044c8bddfb10ad1f2428a: Deleting consensus metadata
I20260923 21:03:41.637341 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.597630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.0.97.1:33713)
0923 21:03:41.597912 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.597922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.598021 (+ 99us) server_negotiation.cc:407] Connection header received
0923 21:03:41.601767 (+ 3746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.601777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.601927 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.602091 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:41.612158 (+ 10067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.613834 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.617097 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.618093 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.618280 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.627037 (+ 8757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:41.627053 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:41.627079 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:41.627170 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:41.632516 (+ 5346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.633585 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.633594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.633598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.633758 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.634457 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.634463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.634466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.634952 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:41.636483 (+ 1531us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.636674 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.636943 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260923 21:03:41.637980 13829 ts_tablet_manager.cc:1918] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:41.638212 13829 log.cc:1196] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/819a5338b5004741a6efb96e39174367
I20260923 21:03:41.638312 13494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:41.638919 13829 ts_tablet_manager.cc:1939] T 819a5338b5004741a6efb96e39174367 P 857f244397674859b0b95b00878a66ff: Deleting consensus metadata
I20260923 21:03:41.639716 13286 catalog_manager.cc:4841] TS 5e8eb31cf83044c8bddfb10ad1f2428a (127.0.97.2:37695): tablet 819a5338b5004741a6efb96e39174367 (table test-table [id=d2330fc8565a49deb6328ee27d42a634]) successfully deleted
I20260923 21:03:41.640944 13285 catalog_manager.cc:4841] TS 857f244397674859b0b95b00878a66ff (127.0.97.3:40325): tablet 819a5338b5004741a6efb96e39174367 (table test-table [id=d2330fc8565a49deb6328ee27d42a634]) successfully deleted
I20260923 21:03:41.639633 13831 ts_tablet_manager.cc:1905] T 819a5338b5004741a6efb96e39174367 P 8f3c8be2a82e4fa6a8670dc7b2a5578f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:41.804556 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 13338
W20260923 21:03:41.823736 13288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33713 (error 108)
I20260923 21:03:41.824600 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 13503
I20260923 21:03:41.826638 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.825629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58522)
0923 21:03:41.826266 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.826464 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":395,"threads_started":1}
W20260923 21:03:41.826952 13288 catalog_manager.cc:4584] TS 8f3c8be2a82e4fa6a8670dc7b2a5578f (127.0.97.1:33713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 819a5338b5004741a6efb96e39174367: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
I20260923 21:03:41.839499 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.825527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.0.97.2:37695)
0923 21:03:41.825718 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.825724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.825745 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:03:41.826030 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.826035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.826101 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.826212 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:41.827664 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.828400 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.829732 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.830199 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.830367 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.833183 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:41.833199 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:41.833217 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:41.833273 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:41.836343 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.837203 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.837212 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.837215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.837367 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.837791 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.837797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.837812 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.838223 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:41.838594 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.838890 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.839127 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 21:03:41.840240 13639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:41.880862 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.880278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58530)
0923 21:03:41.880509 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.880614 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 21:03:41.882176 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.881669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58538)
0923 21:03:41.881917 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.882020 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 21:03:41.933564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 13503
I20260923 21:03:41.942684 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.942200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58542)
0923 21:03:41.942439 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.942540 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 21:03:41.944033 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.943514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58544)
0923 21:03:41.943713 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:41.943821 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 21:03:41.950994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 13648
I20260923 21:03:41.966141 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:41.951807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.0.97.3:40325)
0923 21:03:41.952399 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:41.952404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:41.952421 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:03:41.952497 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:41.952500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:41.952564 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:41.952660 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:41.954052 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.955008 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.956355 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:41.957005 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:41.957136 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:41.959787 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:41.959804 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:41.959821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:41.959931 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:41.963011 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.963797 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.963807 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.963810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.963979 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:41.964341 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:41.964344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:41.964346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:41.964832 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:41.965103 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:41.965426 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 21:03:41.965624 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":298,"threads_started":1}
I20260923 21:03:41.967175 13780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:42.024737 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:42.024160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58546)
0923 21:03:42.024441 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:42.024557 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260923 21:03:42.026048 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:42.025546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33713 (local address 127.0.0.1:58556)
0923 21:03:42.025782 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:42.025887 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260923 21:03:42.059942 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 13648
I20260923 21:03:42.078791 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 13268
I20260923 21:03:42.080381 13331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:42.197759 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 13268
I20260923 21:03:42.224208 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 21:03:42.560818 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 21:03:42.561071 388 mini_ranger.cc:66] Stopping Ranger...
I20260923 21:03:42.907725 388 mini_ranger.cc:68] Stopped Ranger
2026-09-23 21:03:42.908 UTC [12320] LOG: received smart shutdown request
2026-09-23 21:03:42.911 UTC [12320] LOG: background worker "logical replication launcher" (PID 12329) exited with exit code 1
2026-09-23 21:03:42.912 UTC [12324] LOG: shutting down
2026-09-23 21:03:43.128 UTC [12320] LOG: database system is shut down
2026-09-23T21:03:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57566 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:03:43 dist-test-slave-8xn5 krb5kdc[13847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:03:43 dist-test-slave-8xn5 krb5kdc[13847](info): set up 2 sockets
Sep 23 21:03:43 dist-test-slave-8xn5 krb5kdc[13847](info): commencing operation
krb5kdc: starting...
W20260923 21:03:44.483227 388 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:03:44 dist-test-slave-8xn5 krb5kdc[13847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:03:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:03:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:03:44.735035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42209,127.0.97.61:42509,127.0.97.60:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:45.096097 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:45.096495 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:45.096648 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:45.104291 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:45.104485 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:45.104600 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:45.104707 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:45.104787 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:45.115438 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42209,127.0.97.61:42509,127.0.97.60:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:45.117122 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:45.118696 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:45.132115 13866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:45.132190 13865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:45.132212 13868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:45.132946 13860 server_base.cc:1034] running on GCE node
I20260923 21:03:45.133754 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:45.135403 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:45.136621 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790197425136567 us; error 49 us; skew 500 ppm
I20260923 21:03:45.137049 13860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:45 dist-test-slave-8xn5 krb5kdc[13847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:45.140596 13860 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:03:45.142162 13860 webserver.cc:460] Webserver started at http://127.0.97.62:38553/ using document root <none> and password file <none>
I20260923 21:03:45.142951 13860 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:45.143078 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:45.143405 13860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:45.146592 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "894a7b1712cb475ca177c9a458fc1669"
format_stamp: "Formatted at 2026-09-23 21:03:45 on dist-test-slave-8xn5"
server_key: "560f3663a2d65d7c5641d8c528fbf963"
server_key_iv: "66ac7a5a89433b08a49bbdbb2d6f476e"
server_key_version: "encryptionkey@0"
I20260923 21:03:45.147409 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "894a7b1712cb475ca177c9a458fc1669"
format_stamp: "Formatted at 2026-09-23 21:03:45 on dist-test-slave-8xn5"
server_key: "560f3663a2d65d7c5641d8c528fbf963"
server_key_iv: "66ac7a5a89433b08a49bbdbb2d6f476e"
server_key_version: "encryptionkey@0"
I20260923 21:03:45.153436 13860 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 21:03:45.157292 13875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:45.159153 13860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:03:45.159389 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "894a7b1712cb475ca177c9a458fc1669"
format_stamp: "Formatted at 2026-09-23 21:03:45 on dist-test-slave-8xn5"
server_key: "560f3663a2d65d7c5641d8c528fbf963"
server_key_iv: "66ac7a5a89433b08a49bbdbb2d6f476e"
server_key_version: "encryptionkey@0"
I20260923 21:03:45.159637 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:45.184018 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:45.189256 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:45.189594 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:45.209905 13860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42209
I20260923 21:03:45.209939 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42209 every 8 connection(s)
I20260923 21:03:45.211855 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:03:45.216221 13927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:45.219262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 13860
I20260923 21:03:45.219488 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:03:45.220118 388 external_mini_cluster.cc:1428] Setting key 7c251c4988fc77567c6bf2ef02d1d349
I20260923 21:03:45.224154 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 23 21:03:45 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:45.258177 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.228033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.0.97.62:42209)
0923 21:03:45.230242 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:45.230251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:45.230298 (+ 47us) server_negotiation.cc:407] Connection header received
0923 21:03:45.233190 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:45.233197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:45.233509 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:45.233839 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:45.235849 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.237312 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.240093 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.240728 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.240977 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.248192 (+ 7215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:45.248206 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:45.248229 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:45.248289 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:45.252943 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.253801 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.253815 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.253822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.254057 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.254565 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.254572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.254577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.255433 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:45.255684 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:45.256594 (+ 910us) server_negotiation.cc:299] Negotiation successful
0923 21:03:45.256978 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":1988,"spinlock_wait_cycles":59776,"thread_start_us":1736,"threads_started":1}
I20260923 21:03:45.258258 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.228033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42209 (local address 127.0.0.1:42302)
0923 21:03:45.229148 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.229205 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:45.232627 (+ 3422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:45.233918 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:45.233930 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:45.234954 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:45.235568 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.235602 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.237505 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.237514 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.239911 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.239934 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.241109 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.241120 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.241360 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.242381 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:45.242424 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:45.247771 (+ 5347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:45.253141 (+ 5370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.253153 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.253164 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.253618 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.254186 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.254189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.254192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.254424 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.255774 (+ 1350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:45.255788 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:45.256097 (+ 309us) client_negotiation.cc:769] Sending connection context
0923 21:03:45.256442 (+ 345us) client_negotiation.cc:240] Negotiation successful
0923 21:03:45.257193 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":103,"thread_start_us":463,"threads_started":1}
I20260923 21:03:45.262118 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:03:45.263645 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.263017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47394)
0923 21:03:45.263252 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.263442 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 21:03:45.264002 13878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
I20260923 21:03:45.266155 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.265602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47400)
0923 21:03:45.265810 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.265940 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 21:03:45.266572 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 21:03:45.279748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42509
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42209,127.0.97.61:42509,127.0.97.60:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:45.316035 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 1
I20260923 21:03:45.317435 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.316894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47414)
0923 21:03:45.317125 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.317241 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 21:03:45.318900 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.318255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47428)
0923 21:03:45.318608 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.318705 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 21:03:45.319119 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
I20260923 21:03:45.387447 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 2
I20260923 21:03:45.389066 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.388415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47444)
0923 21:03:45.388753 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.388885 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 21:03:45.390264 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.389826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47454)
0923 21:03:45.389984 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.390085 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 21:03:45.390522 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
I20260923 21:03:45.481835 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 3
I20260923 21:03:45.483467 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.482748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47460)
0923 21:03:45.483081 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.483230 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 21:03:45.484804 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.484270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47462)
0923 21:03:45.484474 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.484586 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 21:03:45.485095 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
I20260923 21:03:45.628396 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 4
W20260923 21:03:45.629017 13934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:45.629354 13934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:45.629501 13934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 21:03:45.629806 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.629159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47476)
0923 21:03:45.629540 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.629651 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260923 21:03:45.631067 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.630666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47482)
0923 21:03:45.630814 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.630918 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 21:03:45.631387 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42509: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42509: connect: Connection refused (error 111)
W20260923 21:03:45.637569 13934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:45.637729 13934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:45.637835 13934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:45.637939 13934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:45.638010 13934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:45.648403 13934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42209,127.0.97.61:42509,127.0.97.60:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:45.649978 13934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:45.651424 13934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:45.663875 13942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:45.663960 13939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:45.663987 13940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:45.665022 13934 server_base.cc:1034] running on GCE node
I20260923 21:03:45.666100 13934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:45.667829 13934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:45.669094 13934 hybrid_clock.cc:648] HybridClock initialized: now 1790197425669027 us; error 53 us; skew 500 ppm
I20260923 21:03:45.669523 13934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:45 dist-test-slave-8xn5 krb5kdc[13847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:45.673033 13934 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 21:03:45.674633 13934 webserver.cc:460] Webserver started at http://127.0.97.61:36257/ using document root <none> and password file <none>
I20260923 21:03:45.675362 13934 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:45.675468 13934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:45.675714 13934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:45.679045 13934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data/instance:
uuid: "2a2541b85efb43debf8b18032a249b30"
format_stamp: "Formatted at 2026-09-23 21:03:45 on dist-test-slave-8xn5"
server_key: "4b602133a60ea140143ab6c10c1c12af"
server_key_iv: "aa6ae6736673c980cd6ae6d5667028d7"
server_key_version: "encryptionkey@0"
I20260923 21:03:45.679884 13934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/wal/instance:
uuid: "2a2541b85efb43debf8b18032a249b30"
format_stamp: "Formatted at 2026-09-23 21:03:45 on dist-test-slave-8xn5"
server_key: "4b602133a60ea140143ab6c10c1c12af"
server_key_iv: "aa6ae6736673c980cd6ae6d5667028d7"
server_key_version: "encryptionkey@0"
I20260923 21:03:45.685537 13934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260923 21:03:45.689567 13949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:45.691283 13934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:03:45.691493 13934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/wal
uuid: "2a2541b85efb43debf8b18032a249b30"
format_stamp: "Formatted at 2026-09-23 21:03:45 on dist-test-slave-8xn5"
server_key: "4b602133a60ea140143ab6c10c1c12af"
server_key_iv: "aa6ae6736673c980cd6ae6d5667028d7"
server_key_version: "encryptionkey@0"
I20260923 21:03:45.691709 13934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:45.724732 13934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:45.729526 13934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:45.729862 13934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:45.749729 13934 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42509
I20260923 21:03:45.749765 14000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42509 every 8 connection(s)
I20260923 21:03:45.751268 13934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
I20260923 21:03:45.753639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 13934
I20260923 21:03:45.753837 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-1/wal/instance
I20260923 21:03:45.754267 388 external_mini_cluster.cc:1428] Setting key 614a0b198c248b6a3e109ceb26363885
I20260923 21:03:45.755509 14001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:45.762904 14001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 23 21:03:45 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:45.789516 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.766956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.0.97.62:42209)
0923 21:03:45.767948 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:45.767954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:45.769136 (+ 1182us) server_negotiation.cc:407] Connection header received
0923 21:03:45.772420 (+ 3284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:45.772428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:45.772530 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:45.772636 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:45.775221 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.775994 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.778192 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.778705 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.778864 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.783259 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:45.783264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:45.783268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:45.783317 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:45.786095 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.787020 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.787025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.787028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.787156 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.787638 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.787642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.787645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.787941 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:45.788117 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:45.788886 (+ 769us) server_negotiation.cc:299] Negotiation successful
0923 21:03:45.789122 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":698,"threads_started":1}
I20260923 21:03:45.790628 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.767967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42209 (local address 127.0.0.1:42314)
0923 21:03:45.768901 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.768968 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:45.771991 (+ 3023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:45.772984 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:45.773002 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:45.773936 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:45.774945 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.774977 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.776196 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.776204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.778025 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.778045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.778990 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.778999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.779276 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.780066 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:45.780134 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:45.782984 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:45.786278 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.786289 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.786309 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.786868 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.787276 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.787281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.787285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.787505 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.788099 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:45.788109 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:45.788494 (+ 385us) client_negotiation.cc:769] Sending connection context
0923 21:03:45.788773 (+ 279us) client_negotiation.cc:240] Negotiation successful
0923 21:03:45.789374 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":377,"threads_started":1}
I20260923 21:03:45.792433 14001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:03:45 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 21:03:45.810403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44923
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42209,127.0.97.61:42509,127.0.97.60:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:45.810534 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.793297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47496)
0923 21:03:45.793476 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.793492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:45.793579 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:45.794795 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:45.794799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:45.795258 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:45.795417 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.795425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.796860 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.796866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.797683 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.797693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.798593 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.798599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.798808 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.799406 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:45.799423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:45.803599 (+ 4176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:45.807370 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.807374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.807377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.807802 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.808226 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.808229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.808231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.808479 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.809373 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:45.809377 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:45.809541 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 21:03:45.809709 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 21:03:45.810069 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 21:03:45.810595 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.793628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.0.97.61:42509)
0923 21:03:45.794298 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:45.794303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:45.794405 (+ 102us) server_negotiation.cc:407] Connection header received
0923 21:03:45.794456 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:45.794462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:45.794652 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:45.794813 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:45.795585 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.796715 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.797813 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.798440 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.798576 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.803829 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:45.803842 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:45.803857 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:45.803889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:45.807159 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.807970 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.807978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.807981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.808113 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.808608 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.808612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.808614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.809215 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:45.809435 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:45.809872 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 21:03:45.810200 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":522,"thread_start_us":391,"threads_started":1}
I20260923 21:03:45.813455 14001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:03:45.814673 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.814125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52918)
0923 21:03:45.814314 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.814494 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 21:03:45.814921 13952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:45.816716 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.816270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52922)
0923 21:03:45.816454 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.816556 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 21:03:45.816985 14001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:45.866439 14001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 1
I20260923 21:03:45.867736 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.867168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52926)
0923 21:03:45.867405 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.867540 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 21:03:45.868921 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.868467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52928)
0923 21:03:45.868650 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.868756 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 21:03:45.869185 14001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:45.930644 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } attempt: 5
I20260923 21:03:45.937409 14001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 2
I20260923 21:03:45.938454 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.937951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52936)
0923 21:03:45.938143 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.938225 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
Sep 23 21:03:45 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:03:45.939495 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.939129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52942)
0923 21:03:45.939272 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.939354 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 21:03:45.939693 14001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:45.947086 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.931404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47508)
0923 21:03:45.931660 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.931684 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:45.931825 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:45.932260 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:45.932265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:45.932828 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:45.933070 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.933083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.934103 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.934109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.935463 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:45.935480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.936256 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.936262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:45.936466 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.937147 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:45.937171 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:45.940065 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:45.944207 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.944212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.944215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.944629 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.945167 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:45.945171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:45.945173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:45.945321 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:45.946115 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:45.946119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:45.946277 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 21:03:45.946470 (+ 193us) client_negotiation.cc:240] Negotiation successful
0923 21:03:45.946824 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 21:03:45.947211 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.931532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.0.97.61:42509)
0923 21:03:45.931717 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:45.931722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:45.931750 (+ 28us) server_negotiation.cc:407] Connection header received
0923 21:03:45.932027 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:45.932032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:45.932101 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:45.932206 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:45.933241 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.933946 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.935631 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:45.936101 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:45.936198 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:45.940278 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:45.940284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:45.940289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:45.940329 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:45.944009 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.944803 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.944811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.944816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.945010 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:45.945452 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:45.945459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:45.945465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:45.945957 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:45.946140 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:45.946601 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 21:03:45.946861 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:03:45.948283 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:03:45.949437 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.949023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52958)
0923 21:03:45.949189 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.949278 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 21:03:45.950513 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:45.950116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52970)
0923 21:03:45.950262 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:45.950394 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 21:03:45.950704 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:46.001952 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 1
I20260923 21:03:46.003257 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.002777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52984)
0923 21:03:46.002988 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.003084 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 21:03:46.004240 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.003904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52994)
0923 21:03:46.004063 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.004135 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 21:03:46.004451 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:46.030938 14001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 3
I20260923 21:03:46.032272 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.031681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:52996)
0923 21:03:46.032002 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.032101 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260923 21:03:46.033344 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.032904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53002)
0923 21:03:46.033143 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.033218 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260923 21:03:46.033545 14001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:46.072698 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 2
I20260923 21:03:46.074111 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.073528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53006)
0923 21:03:46.073831 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.073948 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260923 21:03:46.075515 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.075113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53008)
0923 21:03:46.075297 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.075385 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 21:03:46.075728 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
W20260923 21:03:46.169847 14009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:46.170267 14009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:46.170468 14009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 21:03:46.176865 14001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 4
I20260923 21:03:46.178161 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.177607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53018)
0923 21:03:46.177890 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.177989 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 21:03:46.179379 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.178953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53024)
0923 21:03:46.179169 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.179264 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 21:03:46.179481 14009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:46.179672 14009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:46.179617 14001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
W20260923 21:03:46.179780 14009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:46.179889 14009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:46.179979 14009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:46.181962 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 3
I20260923 21:03:46.183156 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.182669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53034)
0923 21:03:46.182880 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.182996 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 21:03:46.184327 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.183922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53036)
0923 21:03:46.184111 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.184194 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 21:03:46.184551 13927 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44923: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44923: connect: Connection refused (error 111)
I20260923 21:03:46.191717 14009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42209,127.0.97.61:42509,127.0.97.60:44923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:46.193336 14009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:46.194980 14009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:46.209306 14017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:46.210577 14009 server_base.cc:1034] running on GCE node
W20260923 21:03:46.214309 14014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:46.214783 14015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:46.215977 14009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:46.218160 14009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:46.219487 14009 hybrid_clock.cc:648] HybridClock initialized: now 1790197426219365 us; error 115 us; skew 500 ppm
I20260923 21:03:46.220028 14009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:46.223943 14009 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 21:03:46.225586 14009 webserver.cc:460] Webserver started at http://127.0.97.60:35887/ using document root <none> and password file <none>
I20260923 21:03:46.226423 14009 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:46.226563 14009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:46.226882 14009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:46.230181 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data/instance:
uuid: "1b29587d6c654f9a865f7666a1eea094"
format_stamp: "Formatted at 2026-09-23 21:03:46 on dist-test-slave-8xn5"
server_key: "0c1e0388ba64f300e50597fefa6a2af4"
server_key_iv: "dcfd5ce87bc5bd72f8d6212880cb4ca8"
server_key_version: "encryptionkey@0"
I20260923 21:03:46.231122 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/wal/instance:
uuid: "1b29587d6c654f9a865f7666a1eea094"
format_stamp: "Formatted at 2026-09-23 21:03:46 on dist-test-slave-8xn5"
server_key: "0c1e0388ba64f300e50597fefa6a2af4"
server_key_iv: "dcfd5ce87bc5bd72f8d6212880cb4ca8"
server_key_version: "encryptionkey@0"
I20260923 21:03:46.237262 14009 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:03:46.241071 14024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:46.242712 14009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 21:03:46.242904 14009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/wal
uuid: "1b29587d6c654f9a865f7666a1eea094"
format_stamp: "Formatted at 2026-09-23 21:03:46 on dist-test-slave-8xn5"
server_key: "0c1e0388ba64f300e50597fefa6a2af4"
server_key_iv: "dcfd5ce87bc5bd72f8d6212880cb4ca8"
server_key_version: "encryptionkey@0"
I20260923 21:03:46.243122 14009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:46.266647 14009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:46.271793 14009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:46.272112 14009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:46.292977 14009 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44923
I20260923 21:03:46.293000 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44923 every 8 connection(s)
I20260923 21:03:46.294535 14009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
I20260923 21:03:46.295855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14009
I20260923 21:03:46.296060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790197207951953-388-0/minicluster-data/master-2/wal/instance
I20260923 21:03:46.296761 388 external_mini_cluster.cc:1428] Setting key 263429a2904ed92acf2fbdd4d04000de
I20260923 21:03:46.299798 14076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:46.307319 14076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:46.321305 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.298919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.0.97.60:44923)
0923 21:03:46.299821 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.299835 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.299915 (+ 80us) server_negotiation.cc:407] Connection header received
0923 21:03:46.301675 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.301700 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.302172 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.302729 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.304387 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.305823 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.307307 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.308108 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.308338 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.313409 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.313417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.313429 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.313470 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.316837 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.317603 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.317616 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.317623 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.317855 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.318266 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.318271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.318275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.318935 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.319261 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.319729 (+ 468us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.320195 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":395,"threads_started":1}
I20260923 21:03:46.328181 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.310717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42209 (local address 127.0.0.1:42324)
0923 21:03:46.311805 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.311860 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:46.312065 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:46.312536 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:46.312542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:46.313297 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:46.313680 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.313692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.315199 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.315206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.316573 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.316591 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.317446 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.317453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.317662 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.318388 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:46.318410 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:46.320762 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:46.324898 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.324911 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.324921 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.325474 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.325876 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.325881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.325885 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.326063 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.326815 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:46.326826 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:46.327070 (+ 244us) client_negotiation.cc:769] Sending connection context
0923 21:03:46.327315 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 21:03:46.327739 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":622,"threads_started":1}
I20260923 21:03:46.328284 14080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.310698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.0.97.62:42209)
0923 21:03:46.311244 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.311250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.311993 (+ 743us) server_negotiation.cc:407] Connection header received
0923 21:03:46.312208 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.312216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.312335 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.312461 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.313866 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.314934 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.316776 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.317236 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.317399 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.320981 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.320986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.320990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.321035 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.324575 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.325577 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.325583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.325586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.325754 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.326183 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.326187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.326189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.326568 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.326685 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.327449 (+ 764us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.327759 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260923 21:03:46.329710 14076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:03:46.343521 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.324794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.0.97.62:42209)
0923 21:03:46.325445 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.325450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.325468 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:03:46.325508 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.325511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.325603 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.325713 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.327611 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.329350 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.330955 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.331628 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.331788 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.336094 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.336100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.336104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.336146 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.339769 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.340815 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.340821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.340825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.340988 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.341473 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.341480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.341484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.341890 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.342198 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.342895 (+ 697us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.343106 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":264,"threads_started":1}
I20260923 21:03:46.344826 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.330564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42509 (local address 127.0.0.1:47514)
0923 21:03:46.330792 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.330817 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:46.330931 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:46.331484 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:46.331488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:46.331873 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:46.332016 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.332024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.333040 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.333045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.333927 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.333938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.334723 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.334729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.334930 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.335505 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:46.335524 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:46.337794 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:46.341795 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.341799 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.341802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.342457 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.342937 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.342946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.342952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.343161 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.344063 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:46.344066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:46.344202 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 21:03:46.344316 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 21:03:46.344524 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 21:03:46.345163 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.330936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.0.97.61:42509)
0923 21:03:46.331139 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.331152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.331169 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:03:46.331236 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.331243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.331330 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.331434 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.332221 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.332885 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.334070 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.334562 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.334707 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.338034 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.338039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.338056 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.338110 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.341623 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.342600 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.342604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.342606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.342798 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.343488 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.343495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.343501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.343831 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.343991 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.344494 (+ 503us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.344783 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:03:46.346266 14076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:03:46.361608 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.347222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53042 (local address 127.0.97.60:44923)
0923 21:03:46.347383 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.347390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.347427 (+ 37us) server_negotiation.cc:407] Connection header received
0923 21:03:46.347472 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.347477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.347540 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.347638 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.348419 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.349572 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.351340 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.351996 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.352130 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.355569 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.355578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.355582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.355620 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.358546 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.359341 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.359345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.359348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.359509 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.359934 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.359941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.359944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.360290 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.360453 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.360902 (+ 449us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.361172 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":59}
I20260923 21:03:46.361756 13927 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 4
I20260923 21:03:46.361677 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.347033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53042)
0923 21:03:46.347225 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.347247 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:46.347363 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:46.347654 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:46.347658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:46.348097 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:46.348269 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.348284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.349811 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.349816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.351154 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.351173 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.352226 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.352272 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.352509 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.353068 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:46.353084 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:46.355383 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:46.358684 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.358690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.358694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.359201 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.359672 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.359677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.359681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.359826 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.360405 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:46.360410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:46.360555 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 21:03:46.360824 (+ 269us) client_negotiation.cc:240] Negotiation successful
0923 21:03:46.361317 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":171}
I20260923 21:03:46.362303 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.345699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.0.97.61:42509)
0923 21:03:46.345882 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.345890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.345909 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:03:46.345958 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.345963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.346026 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.346120 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.347819 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.348546 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.350426 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.351123 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.351295 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.356385 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.356390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.356393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.356428 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.359429 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.360139 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.360145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.360149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.360303 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.360698 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.360704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.360708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.360980 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.361281 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.361524 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.361770 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:03:46.369359 14076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: Bootstrap starting.
I20260923 21:03:46.370193 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 21:03:46.373987 14076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:46.375607 14076 log.cc:824] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:46.381683 14076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: No bootstrap required, opened a new log
I20260923 21:03:46.381824 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.362582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53054)
0923 21:03:46.362770 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.362792 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:46.362912 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:46.363643 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:46.363650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:46.364277 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:46.364464 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.364476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.366404 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.366413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.367896 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.367918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.369021 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.369028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.369215 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.370048 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:46.370082 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:46.373017 (+ 2935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:46.378233 (+ 5216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.378241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.378246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.379006 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.379637 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.379643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.379647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.379903 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.380684 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:46.380691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:46.380879 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 21:03:46.381045 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 21:03:46.381389 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 21:03:46.383050 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.362469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53054 (local address 127.0.97.60:44923)
0923 21:03:46.363170 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.363176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.363199 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:03:46.363328 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.363336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.363431 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.363575 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.364663 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.366228 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.368097 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.368831 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.368981 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.373287 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.373294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.373299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.373340 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.378038 (+ 4698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.379228 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.379234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.379238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.379458 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.380074 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.380079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.380082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.380511 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.382113 (+ 1602us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.382264 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.382535 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":323,"threads_started":1}
I20260923 21:03:46.390990 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669: Bootstrap starting.
I20260923 21:03:46.391973 14076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.392678 14076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:46.392897 14076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b29587d6c654f9a865f7666a1eea094, State: Initialized, Role: FOLLOWER
I20260923 21:03:46.393989 14076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.396648 13927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:46.398322 13927 log.cc:824] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:46.401430 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.404086 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:03:46.403568 14076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:46.409403 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669: No bootstrap required, opened a new log
I20260923 21:03:46.415282 14082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.377461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.0.97.62:42209)
0923 21:03:46.377690 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.377699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.394402 (+ 16703us) server_negotiation.cc:407] Connection header received
0923 21:03:46.394659 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.394664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.394751 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.394870 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.397701 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.398846 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.401628 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.402365 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.402489 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.407098 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.407108 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.407116 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.407156 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.411127 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.412144 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.412149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.412153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.412324 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.412844 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.412850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.412853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.413247 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.413501 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.414648 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.414914 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 21:03:46.421489 13927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.422089 13927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:46.422272 13927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894a7b1712cb475ca177c9a458fc1669, State: Initialized, Role: FOLLOWER
I20260923 21:03:46.422683 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.377055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.0.97.61:42509)
0923 21:03:46.377269 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.377276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.397187 (+ 19911us) server_negotiation.cc:407] Connection header received
0923 21:03:46.399219 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.399230 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.399362 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.399466 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.404205 (+ 4739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.405506 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.408135 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.408903 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.409019 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.413146 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.413155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.413160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.413210 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.418180 (+ 4970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.419483 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.419492 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.419496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.419685 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.420209 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.420217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.420221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.420615 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.420869 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.422014 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.422300 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:03:46.423462 13927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.425758 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.426469 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:03:46.427521 13927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:46.431413 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.398025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.0.97.60:44923)
0923 21:03:46.398270 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.398280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.404348 (+ 6068us) server_negotiation.cc:407] Connection header received
0923 21:03:46.404662 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.404668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.404776 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.404857 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.407422 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.408560 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.410842 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.411659 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.411792 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.421854 (+ 10062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.421865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.421874 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.421976 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.426617 (+ 4641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.427659 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.427668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.427674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.427837 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.428344 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.428349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.428352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.428749 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.429046 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.430591 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.430906 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260923 21:03:46.439266 14110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 21:03:46.439545 14110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260923 21:03:46.442586 13881 tablet.cc:2369] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:46.436035 14009 master_runner.cc:186] Error getting master registration for 127.0.97.62:42209: OK, instance_id { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" instance_seqno: 1790197425204149 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 21:03:46.453260 14009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 21:03:46.456873 14116 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 21:03:46.457170 14116 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 21:03:46.444629 13860 master_runner.cc:186] Error getting master registration for 127.0.97.61:42509: OK, instance_id { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" instance_seqno: 1790197425744137 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 21:03:46.459195 13860 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 21:03:46.478878 14001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } attempt: 5
I20260923 21:03:46.486591 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:03:46.486959 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.489034 14100 leader_election.cc:288] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a2541b85efb43debf8b18032a249b30 (127.0.97.61:42509), 1b29587d6c654f9a865f7666a1eea094 (127.0.97.60:44923)
Sep 23 21:03:46 dist-test-slave-8xn5 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 21:03:46.490357 14051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "894a7b1712cb475ca177c9a458fc1669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b29587d6c654f9a865f7666a1eea094" is_pre_election: true
I20260923 21:03:46.491079 14051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 894a7b1712cb475ca177c9a458fc1669 in term 0.
I20260923 21:03:46.492074 13878 leader_election.cc:302] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b29587d6c654f9a865f7666a1eea094, 894a7b1712cb475ca177c9a458fc1669; no voters:
I20260923 21:03:46.493673 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:03:46.493856 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:03:46.493965 14100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:46.500975 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.503531 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.479756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44923 (local address 127.0.0.1:53070)
0923 21:03:46.480057 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:46.480083 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:46.480228 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:46.480639 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:46.480644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:46.481196 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:46.481416 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.481430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.483915 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.483921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.485442 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:46.485461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.487119 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.487128 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:46.487382 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.488189 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:46.488233 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:46.491494 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:46.499788 (+ 8294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.499797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.499802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.500566 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.501180 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:46.501186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:46.501190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:46.501391 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:46.502286 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:46.502292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:46.502516 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 21:03:46.502689 (+ 173us) client_negotiation.cc:240] Negotiation successful
0923 21:03:46.503078 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260923 21:03:46.504851 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:46.479821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.0.97.60:44923)
0923 21:03:46.480134 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:46.480142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:46.480169 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:03:46.480387 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:46.480394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:46.480486 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:46.480610 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:46.481610 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.482890 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.485629 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:46.486649 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:46.486818 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:46.491827 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:46.491836 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:46.491843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:46.491894 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:46.499527 (+ 7633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.500779 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.500785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.500789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.501010 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:46.501576 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:46.501582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:46.501586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:46.502032 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:46.502197 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:46.504080 (+ 1883us) server_negotiation.cc:299] Negotiation successful
0923 21:03:46.504384 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260923 21:03:46.506461 14100 leader_election.cc:288] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [CANDIDATE]: Term 1 election: Requested vote from peers 2a2541b85efb43debf8b18032a249b30 (127.0.97.61:42509), 1b29587d6c654f9a865f7666a1eea094 (127.0.97.60:44923)
I20260923 21:03:46.507694 14051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "894a7b1712cb475ca177c9a458fc1669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b29587d6c654f9a865f7666a1eea094"
I20260923 21:03:46.507973 14051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:46.511606 14001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30: Bootstrap starting.
I20260923 21:03:46.514537 14051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894a7b1712cb475ca177c9a458fc1669 in term 1.
I20260923 21:03:46.515733 13878 leader_election.cc:302] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b29587d6c654f9a865f7666a1eea094, 894a7b1712cb475ca177c9a458fc1669; no voters:
I20260923 21:03:46.516667 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:03:46.521983 14001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:46.523754 14001 log.cc:824] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30: Log is configured to *not* fsync() on all Append() calls
W20260923 21:03:46.524541 14030 tablet.cc:2369] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:46.526599 14100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [term 1 LEADER]: Becoming Leader. State: Replica: 894a7b1712cb475ca177c9a458fc1669, State: Running, Role: LEADER
I20260923 21:03:46.528357 14100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.531291 14001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30: No bootstrap required, opened a new log
I20260923 21:03:46.545682 14001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.546238 14001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:46.545343 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894a7b1712cb475ca177c9a458fc1669. Latest consensus state: current_term: 1 leader_uuid: "894a7b1712cb475ca177c9a458fc1669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.546538 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:46.546481 14001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2541b85efb43debf8b18032a249b30, State: Initialized, Role: FOLLOWER
I20260923 21:03:46.547667 14001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } }
I20260923 21:03:46.549182 14122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:03:46.552484 14122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:03:46.562304 13975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "894a7b1712cb475ca177c9a458fc1669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a2541b85efb43debf8b18032a249b30"
I20260923 21:03:46.563095 13975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:46.566448 13976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "894a7b1712cb475ca177c9a458fc1669" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a2541b85efb43debf8b18032a249b30" is_pre_election: true
I20260923 21:03:46.569643 13975 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 894a7b1712cb475ca177c9a458fc1669 in term 1.
I20260923 21:03:46.569983 14001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:46.571800 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.572510 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:03:46.577565 14051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [term 1 FOLLOWER]: Refusing update from remote peer 894a7b1712cb475ca177c9a458fc1669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:46.579245 14119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:46.589257 13975 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [term 1 FOLLOWER]: Refusing update from remote peer 894a7b1712cb475ca177c9a458fc1669: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:46.590574 14119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:46.604861 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894a7b1712cb475ca177c9a458fc1669. Latest consensus state: current_term: 1 leader_uuid: "894a7b1712cb475ca177c9a458fc1669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.605388 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:03:46.607900 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894a7b1712cb475ca177c9a458fc1669. Latest consensus state: current_term: 1 leader_uuid: "894a7b1712cb475ca177c9a458fc1669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.608248 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:03:46.615023 14124 mvcc.cc:205] Tried to move back new op lower bound from 7332648659243282432 to 7332648659073888256. Current Snapshot: MvccSnapshot[applied={T|T < 7332648659243282432}]
I20260923 21:03:46.615320 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "894a7b1712cb475ca177c9a458fc1669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.615665 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:03:46.617460 14131 mvcc.cc:205] Tried to move back new op lower bound from 7332648659243282432 to 7332648659073888256. Current Snapshot: MvccSnapshot[applied={T|T < 7332648659243282432 or (T in {7332648659243282432})}]
I20260923 21:03:46.619081 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "894a7b1712cb475ca177c9a458fc1669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.619542 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:46.621850 14119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "894a7b1712cb475ca177c9a458fc1669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.622123 14119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:46.623276 14122 catalog_manager.cc:1347] Generated new cluster ID: d70f9436f92e458ababfe04784ddc520
I20260923 21:03:46.623399 14122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:03:46.628211 14129 mvcc.cc:205] Tried to move back new op lower bound from 7332648659243282432 to 7332648659073888256. Current Snapshot: MvccSnapshot[applied={T|T < 7332648659243282432}]
I20260923 21:03:46.637176 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "894a7b1712cb475ca177c9a458fc1669" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "894a7b1712cb475ca177c9a458fc1669" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42209 } } peers { permanent_uuid: "2a2541b85efb43debf8b18032a249b30" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42509 } } peers { permanent_uuid: "1b29587d6c654f9a865f7666a1eea094" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44923 } } }
I20260923 21:03:46.637490 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:03:46.654524 14122 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:03:46.656528 14122 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:03:46.675848 14122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 894a7b1712cb475ca177c9a458fc1669: Generated new TSK 0
I20260923 21:03:46.677178 14122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:03:46.697917 14143 catalog_manager.cc:1259] Loaded cluster ID: d70f9436f92e458ababfe04784ddc520
I20260923 21:03:46.698134 14143 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30: loading cluster ID for follower catalog manager: success
I20260923 21:03:46.702232 14143 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30: acquiring CA information for follower catalog manager: success
I20260923 21:03:46.705261 14143 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a2541b85efb43debf8b18032a249b30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 21:03:47.441890 14110 catalog_manager.cc:1259] Loaded cluster ID: d70f9436f92e458ababfe04784ddc520
I20260923 21:03:47.442009 14110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: loading cluster ID for follower catalog manager: success
I20260923 21:03:47.444849 14110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: acquiring CA information for follower catalog manager: success
I20260923 21:03:47.446967 14110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b29587d6c654f9a865f7666a1eea094: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 21:03:47.824676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 13860
I20260923 21:03:47.826081 13922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:47.941795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 13860
I20260923 21:03:47.963954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 13934
I20260923 21:03:47.965175 13996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:48.074705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 13934
I20260923 21:03:48.096226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14009
I20260923 21:03:48.097528 14071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:48.206856 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14009
2026-09-23T21:03:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4940 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:03:48 dist-test-slave-8xn5 krb5kdc[14155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:03:48 dist-test-slave-8xn5 krb5kdc[14155](info): set up 2 sockets
Sep 23 21:03:48 dist-test-slave-8xn5 krb5kdc[14155](info): commencing operation
krb5kdc: starting...
W20260923 21:03:49.338902 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:03:49 dist-test-slave-8xn5 krb5kdc[14155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:03:49.587411 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45993
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:49.933985 14168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:49.934311 14168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:49.934502 14168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:49.943481 14168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:49.943610 14168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:49.943665 14168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:49.943724 14168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:49.943768 14168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:49.954437 14168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:49.955811 14168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:49.957199 14168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:49.970366 14174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:49.970454 14173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:49.970430 14176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:49.971741 14168 server_base.cc:1034] running on GCE node
I20260923 21:03:49.972642 14168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:49.974247 14168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:49.975529 14168 hybrid_clock.cc:648] HybridClock initialized: now 1790197429975482 us; error 44 us; skew 500 ppm
I20260923 21:03:49.975904 14168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:49 dist-test-slave-8xn5 krb5kdc[14155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:49.979674 14168 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:03:49.981679 14168 webserver.cc:460] Webserver started at http://127.0.97.62:44685/ using document root <none> and password file <none>
I20260923 21:03:49.982618 14168 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:49.982792 14168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:49.983171 14168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:49.986904 14168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "211f460c525f4798b76e604daed5ab77"
format_stamp: "Formatted at 2026-09-23 21:03:49 on dist-test-slave-8xn5"
server_key: "574a1e0c10cb9263e8abc031fd4fa197"
server_key_iv: "252acc6692a66da05e8ebdc401e59c8c"
server_key_version: "encryptionkey@0"
I20260923 21:03:49.987866 14168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "211f460c525f4798b76e604daed5ab77"
format_stamp: "Formatted at 2026-09-23 21:03:49 on dist-test-slave-8xn5"
server_key: "574a1e0c10cb9263e8abc031fd4fa197"
server_key_iv: "252acc6692a66da05e8ebdc401e59c8c"
server_key_version: "encryptionkey@0"
I20260923 21:03:49.994256 14168 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260923 21:03:49.998613 14183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:50.000196 14168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:03:50.000423 14168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "211f460c525f4798b76e604daed5ab77"
format_stamp: "Formatted at 2026-09-23 21:03:49 on dist-test-slave-8xn5"
server_key: "574a1e0c10cb9263e8abc031fd4fa197"
server_key_iv: "252acc6692a66da05e8ebdc401e59c8c"
server_key_version: "encryptionkey@0"
I20260923 21:03:50.000829 14168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:50.017053 14168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:50.022478 14168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:50.022815 14168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:50.042912 14168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45993
I20260923 21:03:50.042943 14234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45993 every 8 connection(s)
I20260923 21:03:50.044454 14168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:03:50.048413 14235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:50.052093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14168
I20260923 21:03:50.052361 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:03:50.053086 388 external_mini_cluster.cc:1428] Setting key 7d6034263ae1b849c281ea1bd7658bbd
I20260923 21:03:50.061825 14235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77: Bootstrap starting.
I20260923 21:03:50.065764 14235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:50.067209 14235 log.cc:824] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:50.070655 14235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77: No bootstrap required, opened a new log
Sep 23 21:03:50 dist-test-slave-8xn5 krb5kdc[14155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:50.081857 14235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f460c525f4798b76e604daed5ab77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260923 21:03:50.082406 14235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:50.082536 14235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211f460c525f4798b76e604daed5ab77, State: Initialized, Role: FOLLOWER
I20260923 21:03:50.082679 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:50.055232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.0.97.62:45993)
0923 21:03:50.056073 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:50.056091 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:50.056156 (+ 65us) server_negotiation.cc:407] Connection header received
0923 21:03:50.059770 (+ 3614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:50.059805 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:50.060397 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:50.060897 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:50.062579 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:50.064652 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:50.066195 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:50.067386 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:50.067596 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:50.072612 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:50.072626 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:50.072657 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:50.072715 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:50.077111 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:50.078035 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:50.078045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:50.078056 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:50.078290 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:50.078942 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:50.078950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:50.078957 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:50.079863 (+ 906us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:50.080195 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:50.080659 (+ 464us) server_negotiation.cc:299] Negotiation successful
0923 21:03:50.081154 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":349,"threads_started":1}
I20260923 21:03:50.083233 14235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f460c525f4798b76e604daed5ab77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260923 21:03:50.083540 14235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:03:50.083693 14235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:03:50.083882 14235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:50.086656 14235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f460c525f4798b76e604daed5ab77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260923 21:03:50.087072 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:03:50.087231 14235 leader_election.cc:302] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211f460c525f4798b76e604daed5ab77; no voters:
I20260923 21:03:50.087849 14235 leader_election.cc:288] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:03:50.088018 14240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:03:50.088647 14240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [term 1 LEADER]: Becoming Leader. State: Replica: 211f460c525f4798b76e604daed5ab77, State: Running, Role: LEADER
I20260923 21:03:50.089336 14240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f460c525f4798b76e604daed5ab77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } }
I20260923 21:03:50.092144 14235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:50.092342 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211f460c525f4798b76e604daed5ab77. Latest consensus state: current_term: 1 leader_uuid: "211f460c525f4798b76e604daed5ab77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f460c525f4798b76e604daed5ab77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } } }
I20260923 21:03:50.092664 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:50.093331 14242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211f460c525f4798b76e604daed5ab77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211f460c525f4798b76e604daed5ab77" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45993 } } }
I20260923 21:03:50.093640 14242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:50.093653 14244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:03:50.100225 14244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:03:50.112982 14244 catalog_manager.cc:1347] Generated new cluster ID: 638c317a570745289662661ded7ee343
I20260923 21:03:50.113122 14244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:03:50.136801 14244 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:03:50.138240 14244 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:03:50.154475 14244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211f460c525f4798b76e604daed5ab77: Generated new TSK 0
I20260923 21:03:50.155448 14244 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:03:50.201967 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14168
I20260923 21:03:50.203785 14230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:50.309242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14168
2026-09-23T21:03:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2088 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:03:50 dist-test-slave-8xn5 krb5kdc[14263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:03:50 dist-test-slave-8xn5 krb5kdc[14263](info): set up 2 sockets
Sep 23 21:03:50 dist-test-slave-8xn5 krb5kdc[14263](info): commencing operation
krb5kdc: starting...
W20260923 21:03:51.479007 388 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:03:51 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:03:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:03:51.747991 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44601
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:52.090099 14276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:52.090606 14276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:52.090773 14276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:52.098500 14276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:52.098660 14276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:52.098769 14276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:52.098874 14276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:52.098942 14276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:52.109344 14276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:52.110927 14276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:52.112319 14276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:52.124974 14282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:52.125065 14281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:52.125106 14284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:52.125720 14276 server_base.cc:1034] running on GCE node
I20260923 21:03:52.126595 14276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:52.128072 14276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:52.129292 14276 hybrid_clock.cc:648] HybridClock initialized: now 1790197432129243 us; error 35 us; skew 500 ppm
I20260923 21:03:52.129613 14276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:52 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:52.137056 14276 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:03:52.138782 14276 webserver.cc:460] Webserver started at http://127.0.97.62:42451/ using document root <none> and password file <none>
I20260923 21:03:52.139506 14276 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:52.139616 14276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:52.139957 14276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:52.143079 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "70f55b10ad9e4388884c9e03137f2e79"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "cd4a4b485ec5c8f3fe32d06e431b810c"
server_key_iv: "32d5f6bb3be6fca0266e49b3a554c0e3"
server_key_version: "encryptionkey@0"
I20260923 21:03:52.143850 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "70f55b10ad9e4388884c9e03137f2e79"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "cd4a4b485ec5c8f3fe32d06e431b810c"
server_key_iv: "32d5f6bb3be6fca0266e49b3a554c0e3"
server_key_version: "encryptionkey@0"
I20260923 21:03:52.149116 14276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260923 21:03:52.152599 14291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:52.153892 14276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 21:03:52.154070 14276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "70f55b10ad9e4388884c9e03137f2e79"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "cd4a4b485ec5c8f3fe32d06e431b810c"
server_key_iv: "32d5f6bb3be6fca0266e49b3a554c0e3"
server_key_version: "encryptionkey@0"
I20260923 21:03:52.154264 14276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:52.168547 14276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:52.173171 14276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:52.173475 14276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:52.193480 14276 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44601
I20260923 21:03:52.193507 14342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44601 every 8 connection(s)
I20260923 21:03:52.195181 14276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:03:52.199352 14343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:52.203461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14276
I20260923 21:03:52.203688 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:03:52.204502 388 external_mini_cluster.cc:1428] Setting key e760616274efe2d9d418fa446931ab26
I20260923 21:03:52.214987 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Bootstrap starting.
I20260923 21:03:52.219368 14343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:52.220441 14343 log.cc:824] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:03:52 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:52.223884 14343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: No bootstrap required, opened a new log
I20260923 21:03:52.230757 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:52.206843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.0.97.62:44601)
0923 21:03:52.207673 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:52.207686 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:52.207730 (+ 44us) server_negotiation.cc:407] Connection header received
0923 21:03:52.209476 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:52.209509 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:52.210113 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:52.210670 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:52.212734 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:52.214914 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:52.217058 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:52.218200 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:52.218465 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:52.222920 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:52.222929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:52.222945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:52.222987 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:52.226310 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:52.227276 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:52.227282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:52.227289 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:52.227468 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:52.227900 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:52.227904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:52.227905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:52.228434 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:52.228698 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:52.229151 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 21:03:52.229557 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":375,"threads_started":1}
I20260923 21:03:52.232960 14343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:52.233361 14343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:52.233476 14343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f55b10ad9e4388884c9e03137f2e79, State: Initialized, Role: FOLLOWER
I20260923 21:03:52.234120 14343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:52.234382 14343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:03:52.234532 14343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:03:52.234701 14343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:52.236996 14343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:52.237519 14343 leader_election.cc:302] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f55b10ad9e4388884c9e03137f2e79; no voters:
I20260923 21:03:52.238114 14343 leader_election.cc:288] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:03:52.238245 14348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:03:52.239099 14348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 LEADER]: Becoming Leader. State: Replica: 70f55b10ad9e4388884c9e03137f2e79, State: Running, Role: LEADER
I20260923 21:03:52.239614 14348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:52.240372 14343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:52.242573 14349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70f55b10ad9e4388884c9e03137f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } } }
I20260923 21:03:52.243037 14349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:52.243978 14355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:03:52.243816 14350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f55b10ad9e4388884c9e03137f2e79. Latest consensus state: current_term: 1 leader_uuid: "70f55b10ad9e4388884c9e03137f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } } }
I20260923 21:03:52.244203 14350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:52.250180 14355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:03:52.265161 14355 catalog_manager.cc:1347] Generated new cluster ID: 0a95063041be46929c73b51a26339735
I20260923 21:03:52.265309 14355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:03:52.293643 14355 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:03:52.295217 14355 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:03:52.306725 14355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Generated new TSK 0
I20260923 21:03:52.307741 14355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:03:52.367828 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44601
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:52.719599 14369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:52.719942 14369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:52.720121 14369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:52.727864 14369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:52.728035 14369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:52.728223 14369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:03:52.738430 14369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:52.739792 14369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:52.741209 14369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:52.755121 14377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:52.755216 14374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:52.755234 14375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:52.756386 14369 server_base.cc:1034] running on GCE node
I20260923 21:03:52.757040 14369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:52.758183 14369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:52.759394 14369 hybrid_clock.cc:648] HybridClock initialized: now 1790197432759343 us; error 78 us; skew 500 ppm
I20260923 21:03:52.759699 14369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:52 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:52.763196 14369 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:03:52.764966 14369 webserver.cc:460] Webserver started at http://127.0.97.1:42137/ using document root <none> and password file <none>
I20260923 21:03:52.765725 14369 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:52.765873 14369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:52.766191 14369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:52.769280 14369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "0abee61659ed40309cf450b131da4867"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "891b11d444f56756ddda660fbb17286e"
server_key_iv: "fea98379e9fbfbf31a76acd418944c82"
server_key_version: "encryptionkey@0"
I20260923 21:03:52.770121 14369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0abee61659ed40309cf450b131da4867"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "891b11d444f56756ddda660fbb17286e"
server_key_iv: "fea98379e9fbfbf31a76acd418944c82"
server_key_version: "encryptionkey@0"
I20260923 21:03:52.776305 14369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 21:03:52.780221 14384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:52.781670 14369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:03:52.781868 14369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "0abee61659ed40309cf450b131da4867"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "891b11d444f56756ddda660fbb17286e"
server_key_iv: "fea98379e9fbfbf31a76acd418944c82"
server_key_version: "encryptionkey@0"
I20260923 21:03:52.782089 14369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:52.803977 14369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:52.808782 14369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:52.809100 14369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:52.810109 14369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:52.811589 14369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:52.811691 14369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:52.811841 14369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:52.811913 14369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:52.845849 14369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36223
I20260923 21:03:52.845907 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36223 every 8 connection(s)
I20260923 21:03:52.847185 14369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:03:52.855647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14369
I20260923 21:03:52.855867 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:03:52.856523 388 external_mini_cluster.cc:1428] Setting key a3313bfe6edf4d7cf7f04c25913d0244
Sep 23 21:03:52 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:52.869306 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:52.849730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59579 (local address 127.0.97.62:44601)
0923 21:03:52.850201 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:52.850207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:52.851295 (+ 1088us) server_negotiation.cc:407] Connection header received
0923 21:03:52.853097 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:52.853101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:52.853168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:52.853268 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:52.854930 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:52.855702 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:52.857512 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:52.858032 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:52.858148 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:52.861875 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:52.861885 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:52.861892 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:52.861945 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:52.865797 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:52.866824 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:52.866829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:52.866832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:52.866954 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:52.867398 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:52.867407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:52.867409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:52.867768 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:52.867925 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:52.868850 (+ 925us) server_negotiation.cc:299] Negotiation successful
0923 21:03:52.869032 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260923 21:03:52.870980 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:52.850317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:59579)
0923 21:03:52.851114 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:52.851166 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:52.852836 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:52.853479 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:52.853488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:52.854086 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:52.854764 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:52.854783 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:52.855865 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:52.855875 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:52.857305 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:52.857335 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:52.858269 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:52.858277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:52.858555 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:52.859333 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:52.859361 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:52.861613 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:52.865996 (+ 4383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:52.866005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:52.866020 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:52.866677 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:52.867078 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:52.867087 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:52.867091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:52.867275 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:52.867898 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:52.867906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:52.868412 (+ 506us) client_negotiation.cc:769] Sending connection context
0923 21:03:52.868729 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 21:03:52.869368 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":287,"threads_started":1}
I20260923 21:03:52.872617 14497 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44601
I20260923 21:03:52.872941 14497 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:52.873677 14497 heartbeater.cc:511] Master 127.0.97.62:44601 requested a full tablet report, sending...
I20260923 21:03:52.875911 14308 ts_manager.cc:194] Registered new tserver with Master: 0abee61659ed40309cf450b131da4867 (127.0.97.1:36223)
I20260923 21:03:52.877416 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59579
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:03:52.914561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44601
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:53.268183 14503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:53.268487 14503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:53.268584 14503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:53.275957 14503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:53.276091 14503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:53.276221 14503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:03:53.286559 14503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:53.288568 14503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:53.290189 14503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:53.309227 14508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:53.309273 14509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:53.309574 14511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:53.312345 14503 server_base.cc:1034] running on GCE node
I20260923 21:03:53.313249 14503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:53.314427 14503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:53.315851 14503 hybrid_clock.cc:648] HybridClock initialized: now 1790197433315788 us; error 54 us; skew 500 ppm
I20260923 21:03:53.316214 14503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:53 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:53.319891 14503 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:03:53.322106 14503 webserver.cc:460] Webserver started at http://127.0.97.2:42131/ using document root <none> and password file <none>
I20260923 21:03:53.323069 14503 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:53.323235 14503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:53.323632 14503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:53.327755 14503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "db64176508ee4dae92ba5d20f0dd7fd1"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "5c25dd94422135b8302409adf252635a"
server_key_iv: "81c5cdfca3c911d1e42da2962376da7b"
server_key_version: "encryptionkey@0"
I20260923 21:03:53.328723 14503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "db64176508ee4dae92ba5d20f0dd7fd1"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "5c25dd94422135b8302409adf252635a"
server_key_iv: "81c5cdfca3c911d1e42da2962376da7b"
server_key_version: "encryptionkey@0"
I20260923 21:03:53.335273 14503 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:03:53.339617 14518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:53.341629 14503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:03:53.341830 14503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "db64176508ee4dae92ba5d20f0dd7fd1"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "5c25dd94422135b8302409adf252635a"
server_key_iv: "81c5cdfca3c911d1e42da2962376da7b"
server_key_version: "encryptionkey@0"
I20260923 21:03:53.342087 14503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:53.360253 14503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:53.365896 14503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:53.366225 14503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:53.367395 14503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:53.368844 14503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:53.368952 14503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:53.369120 14503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:53.369202 14503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:53.406646 14503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46123
I20260923 21:03:53.406735 14630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46123 every 8 connection(s)
I20260923 21:03:53.407990 14503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:03:53.410362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14503
I20260923 21:03:53.410635 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:03:53.411237 388 external_mini_cluster.cc:1428] Setting key 760ff7be680b1f921a0e2387d8784970
Sep 23 21:03:53 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:53.436537 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:53.411034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51867 (local address 127.0.97.62:44601)
0923 21:03:53.411523 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:53.411528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:53.412835 (+ 1307us) server_negotiation.cc:407] Connection header received
0923 21:03:53.415660 (+ 2825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:53.415664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:53.415730 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:53.415824 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:53.418165 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.418980 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.421672 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.422219 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.422412 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:53.427319 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:53.427329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:53.427337 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:53.427387 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:53.431973 (+ 4586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:53.433032 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:53.433042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:53.433049 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:53.433267 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:53.433919 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:53.433927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:53.433931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:53.434511 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:53.434718 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:53.435517 (+ 799us) server_negotiation.cc:299] Negotiation successful
0923 21:03:53.435847 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
I20260923 21:03:53.437036 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:53.411712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:51867)
0923 21:03:53.412657 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:53.412713 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:53.415308 (+ 2595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:53.416141 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:53.416157 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:53.417023 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:53.417940 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:53.417969 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.419155 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.419164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:53.421481 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:53.421501 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.422692 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.422699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:53.423091 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:53.424172 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:53.424204 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:53.426987 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:53.432193 (+ 5206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:53.432210 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:53.432228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:53.432850 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:53.433432 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:53.433436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:53.433439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:53.433766 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:53.434670 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:53.434676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:53.435075 (+ 399us) client_negotiation.cc:769] Sending connection context
0923 21:03:53.435333 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 21:03:53.435798 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":389,"threads_started":1}
I20260923 21:03:53.438952 14631 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44601
I20260923 21:03:53.439358 14631 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:03:53.440501 14631 heartbeater.cc:511] Master 127.0.97.62:44601 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260923 21:03:53.442834 14308 ts_manager.cc:194] Registered new tserver with Master: db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2:46123)
I20260923 21:03:53.444605 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:03:53.474468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44601
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:53.822119 14637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:53.822537 14637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:53.822729 14637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:53.830252 14637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:53.830469 14637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:53.830651 14637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:03:53.841358 14637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:53.842859 14637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:53.844309 14637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:53.858511 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:53.858578 14642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:53.858578 14643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:53.860610 14637 server_base.cc:1034] running on GCE node
I20260923 21:03:53.861316 14637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:53.862479 14637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:53.863775 14637 hybrid_clock.cc:648] HybridClock initialized: now 1790197433863712 us; error 46 us; skew 500 ppm
I20260923 21:03:53.864189 14637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:53 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:53.867928 14637 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:03:53.869650 14637 webserver.cc:460] Webserver started at http://127.0.97.3:34123/ using document root <none> and password file <none>
I20260923 21:03:53.870563 14637 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:53.870724 14637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:53.871058 14637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:03:53.874451 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "9aceb9d6306048389443d0e34a05848b"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "0e379526ebaf6da5d07c236123b016ef"
server_key_iv: "bbc145666c708ab631e8c542c4b98973"
server_key_version: "encryptionkey@0"
I20260923 21:03:53.875375 14637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9aceb9d6306048389443d0e34a05848b"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "0e379526ebaf6da5d07c236123b016ef"
server_key_iv: "bbc145666c708ab631e8c542c4b98973"
server_key_version: "encryptionkey@0"
I20260923 21:03:53.880627 14497 heartbeater.cc:503] Master 127.0.97.62:44601 was elected leader, sending a full tablet report...
I20260923 21:03:53.881671 14637 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 21:03:53.886268 14652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:53.888062 14637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:03:53.888262 14637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "9aceb9d6306048389443d0e34a05848b"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "0e379526ebaf6da5d07c236123b016ef"
server_key_iv: "bbc145666c708ab631e8c542c4b98973"
server_key_version: "encryptionkey@0"
I20260923 21:03:53.888499 14637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:53.904906 14637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:53.909688 14637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:53.910053 14637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:53.911105 14637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:53.912496 14637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:53.912582 14637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:53.912683 14637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:53.912725 14637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:53.949311 14637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43187
I20260923 21:03:53.949358 14764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43187 every 8 connection(s)
I20260923 21:03:53.950980 14637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:03:53.961016 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14637
I20260923 21:03:53.961205 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:03:53.961663 388 external_mini_cluster.cc:1428] Setting key 241dbf0cc185478ffa56094b099a3cc5
Sep 23 21:03:53 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:53.975716 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:53.953867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34273 (local address 127.0.97.62:44601)
0923 21:03:53.954618 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:53.954627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:53.955949 (+ 1322us) server_negotiation.cc:407] Connection header received
0923 21:03:53.958667 (+ 2718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:53.958672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:53.958823 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:53.958927 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:53.960977 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.961955 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.964379 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.965115 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.965285 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:53.969423 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:53.969431 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:53.969436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:53.969483 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:53.972234 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:53.973142 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:53.973146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:53.973149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:53.973275 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:53.973762 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:53.973772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:53.973774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:53.974062 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:53.974236 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:53.975075 (+ 839us) server_negotiation.cc:299] Negotiation successful
0923 21:03:53.975323 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":298,"threads_started":1}
I20260923 21:03:53.976797 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:53.954488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:34273)
0923 21:03:53.955725 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:53.955795 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:53.958247 (+ 2452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:53.959215 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:53.959225 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:53.960016 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:53.960740 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:53.960772 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.962120 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.962127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:53.964190 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:53.964203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:53.965414 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:53.965421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:53.965793 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:53.966639 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:53.966668 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:53.969144 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:53.972415 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:53.972427 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:53.972453 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:53.972953 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:53.973396 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:53.973414 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:53.973421 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:53.973633 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:53.974199 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:53.974209 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:53.974668 (+ 459us) client_negotiation.cc:769] Sending connection context
0923 21:03:53.974954 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 21:03:53.975557 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":419,"threads_started":1}
I20260923 21:03:53.978513 14765 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44601
I20260923 21:03:53.978915 14765 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:53.979710 14765 heartbeater.cc:511] Master 127.0.97.62:44601 requested a full tablet report, sending...
I20260923 21:03:53.981340 14308 ts_manager.cc:194] Registered new tserver with Master: 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187)
I20260923 21:03:53.982162 14308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34273
I20260923 21:03:53.991858 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:03:54.010907 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:53.997005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.0.97.62:44601)
0923 21:03:53.997222 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:53.997226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:53.997628 (+ 402us) server_negotiation.cc:407] Connection header received
0923 21:03:53.997776 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:53.997780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:53.997845 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:53.997951 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:53.999617 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.000386 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.001930 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.002547 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.002653 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.005044 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:54.005049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:54.005053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:54.005090 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:54.008101 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.008824 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.008828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.008830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.008951 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.009357 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.009360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.009361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.009705 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:54.009902 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.010318 (+ 416us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.010647 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 21:03:54.016083 14308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:03:54.029716 14308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:03:54.068383 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.045356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.97.2:46123)
0923 21:03:54.052300 (+ 6944us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.052314 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.055466 (+ 3152us) server_negotiation.cc:407] Connection header received
0923 21:03:54.056029 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.056043 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.056435 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.056658 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:54.057969 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.060092 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.064213 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.066299 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.066477 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.066745 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.067625 (+ 880us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.067885 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6748,"thread_start_us":6617,"threads_started":1}
I20260923 21:03:54.069965 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.044777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46123 (local address 127.0.0.1:59452)
0923 21:03:54.055215 (+ 10438us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.055280 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:54.055704 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:54.057095 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:54.057106 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:54.057155 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:54.057732 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.057746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.060347 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.060355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.063988 (+ 3633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.064011 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.066970 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.066978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.067271 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.067342 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 21:03:54.068984 (+ 1642us) client_negotiation.cc:240] Negotiation successful
0923 21:03:54.069316 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9996,"mutex_wait_us":1,"thread_start_us":12521,"threads_started":1}
I20260923 21:03:54.074725 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.046131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.0.97.3:43187)
0923 21:03:54.053452 (+ 7321us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.053465 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.058364 (+ 4899us) server_negotiation.cc:407] Connection header received
0923 21:03:54.059889 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.059899 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.060240 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.060477 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:54.061683 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.065370 (+ 3687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.070388 (+ 5018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.073427 (+ 3039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.073586 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.073837 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.074181 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.074384 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7077,"thread_start_us":6868,"threads_started":1}
I20260923 21:03:54.074733 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.045682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43187 (local address 127.0.0.1:36268)
0923 21:03:54.057653 (+ 11971us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.057676 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:54.059629 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:54.060815 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:54.060823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:54.060857 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:54.061431 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.061450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.065605 (+ 4155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.065614 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.070091 (+ 4477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.070114 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.073689 (+ 3575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.073699 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.073959 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.074004 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 21:03:54.074112 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 21:03:54.074256 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11793,"mutex_wait_us":38,"thread_start_us":8914,"threads_started":1}
I20260923 21:03:54.075964 14566 tablet_service.cc:1467] Processing CreateTablet for tablet af32a3e11abc4006b6b09815c605602c (DEFAULT_TABLE table=test-table [id=f155acc0c9284fe6828f832fe3db2dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:54.077636 14566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af32a3e11abc4006b6b09815c605602c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:54.079273 14700 tablet_service.cc:1467] Processing CreateTablet for tablet af32a3e11abc4006b6b09815c605602c (DEFAULT_TABLE table=test-table [id=f155acc0c9284fe6828f832fe3db2dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:54.080852 14700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af32a3e11abc4006b6b09815c605602c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:54.081223 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.064027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45306)
0923 21:03:54.070519 (+ 6492us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.070544 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:54.070674 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:54.075350 (+ 4676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:54.075355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:54.075383 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:54.075553 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.075564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.077155 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.077160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.078766 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.078787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.080354 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.080358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.080483 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.080520 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 21:03:54.080629 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 21:03:54.080813 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6348,"thread_start_us":6886,"threads_started":1}
I20260923 21:03:54.081244 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.058029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.0.97.1:36223)
0923 21:03:54.074301 (+ 16272us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.074309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.074550 (+ 241us) server_negotiation.cc:407] Connection header received
0923 21:03:54.074661 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.074670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.074985 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.075207 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:54.075716 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.077013 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.078937 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.079870 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.080092 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.080281 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.080687 (+ 406us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.080862 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16077,"thread_start_us":16311,"threads_started":1}
I20260923 21:03:54.084728 14432 tablet_service.cc:1467] Processing CreateTablet for tablet af32a3e11abc4006b6b09815c605602c (DEFAULT_TABLE table=test-table [id=f155acc0c9284fe6828f832fe3db2dac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:54.086869 14432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af32a3e11abc4006b6b09815c605602c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:54.103502 14785 tablet_bootstrap.cc:492] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Bootstrap starting.
I20260923 21:03:54.103739 14784 tablet_bootstrap.cc:492] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Bootstrap starting.
I20260923 21:03:54.107911 14784 tablet_bootstrap.cc:654] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:54.109387 14784 log.cc:824] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:54.109966 14785 tablet_bootstrap.cc:654] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:54.111363 14785 log.cc:824] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:54.115067 14784 tablet_bootstrap.cc:492] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: No bootstrap required, opened a new log
I20260923 21:03:54.115463 14784 ts_tablet_manager.cc:1397] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 21:03:54.119072 14785 tablet_bootstrap.cc:492] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: No bootstrap required, opened a new log
I20260923 21:03:54.119387 14785 ts_tablet_manager.cc:1397] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260923 21:03:54.127018 14784 raft_consensus.cc:348] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.127581 14784 raft_consensus.cc:374] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:54.127710 14784 raft_consensus.cc:729] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db64176508ee4dae92ba5d20f0dd7fd1, State: Initialized, Role: FOLLOWER
I20260923 21:03:54.128504 14787 tablet_bootstrap.cc:492] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Bootstrap starting.
I20260923 21:03:54.128566 14784 consensus_queue.cc:260] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.133790 14787 tablet_bootstrap.cc:654] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:54.135689 14787 log.cc:824] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:54.136641 14785 raft_consensus.cc:348] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.137138 14784 ts_tablet_manager.cc:1428] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
I20260923 21:03:54.137329 14785 raft_consensus.cc:374] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:54.137518 14785 raft_consensus.cc:729] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aceb9d6306048389443d0e34a05848b, State: Initialized, Role: FOLLOWER
I20260923 21:03:54.137784 14631 heartbeater.cc:503] Master 127.0.97.62:44601 was elected leader, sending a full tablet report...
I20260923 21:03:54.138417 14785 consensus_queue.cc:260] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.147802 14765 heartbeater.cc:503] Master 127.0.97.62:44601 was elected leader, sending a full tablet report...
I20260923 21:03:54.148779 14785 ts_tablet_manager.cc:1428] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Time spent starting tablet: real 0.029s user 0.015s sys 0.008s
I20260923 21:03:54.151634 14787 tablet_bootstrap.cc:492] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: No bootstrap required, opened a new log
I20260923 21:03:54.151986 14787 ts_tablet_manager.cc:1397] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.009s
W20260923 21:03:54.160405 14632 tablet.cc:2369] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:54.167199 14787 raft_consensus.cc:348] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.167881 14787 raft_consensus.cc:374] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:54.168008 14787 raft_consensus.cc:729] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0abee61659ed40309cf450b131da4867, State: Initialized, Role: FOLLOWER
I20260923 21:03:54.168900 14787 consensus_queue.cc:260] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.171309 14787 ts_tablet_manager.cc:1428] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
W20260923 21:03:54.202937 14766 tablet.cc:2369] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:54.204066 14789 raft_consensus.cc:482] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:03:54.204365 14789 raft_consensus.cc:504] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.206131 14789 leader_election.cc:288] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0abee61659ed40309cf450b131da4867 (127.0.97.1:36223), 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187)
I20260923 21:03:54.228488 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.207857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50919 (local address 127.0.97.1:36223)
0923 21:03:54.208117 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.208125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.218471 (+ 10346us) server_negotiation.cc:407] Connection header received
0923 21:03:54.219111 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.219121 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.219228 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.219308 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:54.220535 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.222022 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.225578 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.227107 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.227228 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.227335 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.228035 (+ 700us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.228195 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 21:03:54.229552 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.206529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.97.2:50919)
0923 21:03:54.218277 (+ 11748us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.218307 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:54.218876 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:54.219633 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:54.219641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:54.219678 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:54.220303 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.220328 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.222279 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.222287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.225293 (+ 3006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.225320 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.227552 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.227562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.227778 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.227830 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 21:03:54.228960 (+ 1130us) client_negotiation.cc:240] Negotiation successful
0923 21:03:54.229109 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11544,"mutex_wait_us":1,"thread_start_us":11959,"threads_started":1}
I20260923 21:03:54.230935 14452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af32a3e11abc4006b6b09815c605602c" candidate_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0abee61659ed40309cf450b131da4867" is_pre_election: true
I20260923 21:03:54.231660 14452 raft_consensus.cc:2454] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db64176508ee4dae92ba5d20f0dd7fd1 in term 0.
I20260923 21:03:54.232554 14522 leader_election.cc:302] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0abee61659ed40309cf450b131da4867, db64176508ee4dae92ba5d20f0dd7fd1; no voters:
I20260923 21:03:54.233067 14789 raft_consensus.cc:2781] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:03:54.233246 14789 raft_consensus.cc:482] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:03:54.233351 14789 raft_consensus.cc:3037] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:54.234072 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.206795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43187 (local address 127.0.97.2:33765)
0923 21:03:54.226850 (+ 20055us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.226873 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:54.227096 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:54.227626 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:54.227631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:54.227654 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:54.228063 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.228075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.229549 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.229553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.231138 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:54.231150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.233117 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.233123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:54.233326 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.233378 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 21:03:54.233537 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 21:03:54.233696 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19914,"thread_start_us":19542,"threads_started":1}
I20260923 21:03:54.234130 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.207390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33765 (local address 127.0.97.3:43187)
0923 21:03:54.207643 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.207651 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.226976 (+ 19325us) server_negotiation.cc:407] Connection header received
0923 21:03:54.227269 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.227275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.227361 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.227427 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:54.228234 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.229311 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.231312 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.232718 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.232845 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.232947 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.233615 (+ 668us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.233816 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 21:03:54.235273 14720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af32a3e11abc4006b6b09815c605602c" candidate_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aceb9d6306048389443d0e34a05848b" is_pre_election: true
I20260923 21:03:54.236084 14720 raft_consensus.cc:2454] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db64176508ee4dae92ba5d20f0dd7fd1 in term 0.
I20260923 21:03:54.236922 14789 raft_consensus.cc:504] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.237680 14789 leader_election.cc:288] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [CANDIDATE]: Term 1 election: Requested vote from peers 0abee61659ed40309cf450b131da4867 (127.0.97.1:36223), 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187)
I20260923 21:03:54.238229 14720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af32a3e11abc4006b6b09815c605602c" candidate_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aceb9d6306048389443d0e34a05848b"
I20260923 21:03:54.238238 14452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af32a3e11abc4006b6b09815c605602c" candidate_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0abee61659ed40309cf450b131da4867"
I20260923 21:03:54.238454 14720 raft_consensus.cc:3037] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:54.238611 14452 raft_consensus.cc:3037] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:54.240865 14720 raft_consensus.cc:2454] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db64176508ee4dae92ba5d20f0dd7fd1 in term 1.
I20260923 21:03:54.241333 14519 leader_election.cc:302] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aceb9d6306048389443d0e34a05848b, db64176508ee4dae92ba5d20f0dd7fd1; no voters:
I20260923 21:03:54.241729 14789 raft_consensus.cc:2781] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:03:54.242269 14789 raft_consensus.cc:686] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 LEADER]: Becoming Leader. State: Replica: db64176508ee4dae92ba5d20f0dd7fd1, State: Running, Role: LEADER
I20260923 21:03:54.242861 14789 consensus_queue.cc:237] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } }
I20260923 21:03:54.243148 14452 raft_consensus.cc:2454] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db64176508ee4dae92ba5d20f0dd7fd1 in term 1.
I20260923 21:03:54.249042 14307 catalog_manager.cc:5495] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } health_report { overall_health: UNKNOWN } } }
I20260923 21:03:54.273612 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.267765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.0.97.2:46123)
0923 21:03:54.267970 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.267976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.268152 (+ 176us) server_negotiation.cc:407] Connection header received
0923 21:03:54.268403 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.268408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.268489 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.268660 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:03:54.269066 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.269778 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.271392 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.271803 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.271941 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.272467 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:03:54.272997 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:03:54.273186 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.273235 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.273321 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 21:03:54.284873 14720 raft_consensus.cc:1261] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Refusing update from remote peer db64176508ee4dae92ba5d20f0dd7fd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:54.285250 14452 raft_consensus.cc:1261] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Refusing update from remote peer db64176508ee4dae92ba5d20f0dd7fd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:54.285998 14795 consensus_queue.cc:1035] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:54.286517 14789 consensus_queue.cc:1035] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:54.306196 14798 mvcc.cc:205] Tried to move back new op lower bound from 7332648690818965504 to 7332648690663866368. Current Snapshot: MvccSnapshot[applied={T|T < 7332648690818965504}]
I20260923 21:03:54.323709 14308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:03:54.324051 14308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:03:54.329011 14308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Sending DeleteTablet for 3 replicas of tablet af32a3e11abc4006b6b09815c605602c
I20260923 21:03:54.330183 14566 tablet_service.cc:1514] Processing DeleteTablet for tablet af32a3e11abc4006b6b09815c605602c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59452
I20260923 21:03:54.332468 14432 tablet_service.cc:1514] Processing DeleteTablet for tablet af32a3e11abc4006b6b09815c605602c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45306
I20260923 21:03:54.332988 14431 tablet_service.cc:1514] Processing DeleteTablet for tablet af32a3e11abc4006b6b09815c605602c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45306
I20260923 21:03:54.333490 14700 tablet_service.cc:1514] Processing DeleteTablet for tablet af32a3e11abc4006b6b09815c605602c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:54 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36268
I20260923 21:03:54.335232 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14369
I20260923 21:03:54.347512 14807 tablet_replica.cc:330] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: stopping tablet replica
I20260923 21:03:54.348627 14807 raft_consensus.cc:2229] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:03:54.349434 14807 raft_consensus.cc:2258] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:54.355751 14807 ts_tablet_manager.cc:1905] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:54.361549 14809 tablet_replica.cc:330] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: stopping tablet replica
I20260923 21:03:54.364982 14809 raft_consensus.cc:2229] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:03:54.365690 14809 raft_consensus.cc:2258] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:54.369465 14809 ts_tablet_manager.cc:1905] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:54.376264 14807 ts_tablet_manager.cc:1918] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:54.376474 14807 log.cc:1196] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/af32a3e11abc4006b6b09815c605602c
I20260923 21:03:54.377043 14807 ts_tablet_manager.cc:1939] T af32a3e11abc4006b6b09815c605602c P db64176508ee4dae92ba5d20f0dd7fd1: Deleting consensus metadata
I20260923 21:03:54.379076 14295 catalog_manager.cc:4841] TS db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2:46123): tablet af32a3e11abc4006b6b09815c605602c (table test-table [id=f155acc0c9284fe6828f832fe3db2dac]) successfully deleted
Sep 23 21:03:54 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:03:54.383272 14808 tablet_replica.cc:330] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: stopping tablet replica
I20260923 21:03:54.383855 14808 raft_consensus.cc:2229] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:03:54.384403 14808 raft_consensus.cc:2258] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:54.386797 14808 ts_tablet_manager.cc:1905] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:54.388923 14809 ts_tablet_manager.cc:1918] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:54.389117 14809 log.cc:1196] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/af32a3e11abc4006b6b09815c605602c
I20260923 21:03:54.389863 14809 ts_tablet_manager.cc:1939] T af32a3e11abc4006b6b09815c605602c P 9aceb9d6306048389443d0e34a05848b: Deleting consensus metadata
I20260923 21:03:54.393255 14292 catalog_manager.cc:4841] TS 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187): tablet af32a3e11abc4006b6b09815c605602c (table test-table [id=f155acc0c9284fe6828f832fe3db2dac]) successfully deleted
I20260923 21:03:54.394601 14808 ts_tablet_manager.cc:1918] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:54.394774 14808 log.cc:1196] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/af32a3e11abc4006b6b09815c605602c
I20260923 21:03:54.395169 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.336889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.0.97.1:36223)
0923 21:03:54.337133 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.337148 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.360753 (+ 23605us) server_negotiation.cc:407] Connection header received
0923 21:03:54.362048 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.362073 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.362199 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.362282 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:54.366514 (+ 4232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.367772 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.376613 (+ 8841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.377441 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.377569 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.385668 (+ 8099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:54.385687 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:54.385706 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:54.385756 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:54.390579 (+ 4823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.391493 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.391505 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.391508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.391715 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.392302 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.392307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.392323 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.392872 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:54.393266 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.394290 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.394626 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 21:03:54.395792 14808 ts_tablet_manager.cc:1939] T af32a3e11abc4006b6b09815c605602c P 0abee61659ed40309cf450b131da4867: Deleting consensus metadata
I20260923 21:03:54.396075 14492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:54.508076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14369
W20260923 21:03:54.527635 14295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36223 (error 108)
I20260923 21:03:54.527912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14503
I20260923 21:03:54.529546 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.529022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45334)
0923 21:03:54.529291 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.529388 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
W20260923 21:03:54.529876 14295 catalog_manager.cc:4584] TS 0abee61659ed40309cf450b131da4867 (127.0.97.1:36223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af32a3e11abc4006b6b09815c605602c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Sep 23 21:03:54 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:03:54.548796 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.529087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.0.97.2:46123)
0923 21:03:54.529555 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.529562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.529582 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:03:54.529655 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.529661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.529749 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.529874 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:54.531383 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.532821 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.534179 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.535184 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.535330 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.540027 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:54.540047 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:54.540070 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:54.540167 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:54.545137 (+ 4970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.545958 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.545970 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.545975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.546241 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.546782 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.546788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.546792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.547346 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:54.547730 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.548108 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.548362 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260923 21:03:54.549666 14626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:54.574791 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.574187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45344)
0923 21:03:54.574476 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.574600 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 21:03:54.576267 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.575777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45348)
0923 21:03:54.575964 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.576095 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 21:03:54.583714 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.583210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45364)
0923 21:03:54.583443 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.583548 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 21:03:54.584825 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.584393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45366)
0923 21:03:54.584570 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.584657 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 21:03:54.625522 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.624922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45370)
0923 21:03:54.625196 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.625319 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260923 21:03:54.626926 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.626417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45382)
0923 21:03:54.626665 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.626768 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260923 21:03:54.661787 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.661164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45396)
0923 21:03:54.661475 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.661588 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260923 21:03:54.662984 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.662566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45398)
0923 21:03:54.662732 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.662819 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 21:03:54.674131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14503
I20260923 21:03:54.690181 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14637
Sep 23 21:03:54 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:03:54.708527 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.691207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.0.97.3:43187)
0923 21:03:54.691392 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:54.691397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:54.691415 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:03:54.691568 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:54.691572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:54.691673 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:54.691783 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:54.693838 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.694764 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.696365 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:54.696836 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:54.697006 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:54.702305 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:54.702371 (+ 66us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:54.702404 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:54.702451 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:54.705421 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.706221 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.706229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.706231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.706441 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:54.706856 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:54.706861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:54.706865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:54.707252 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:54.707549 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:54.707898 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 21:03:54.708137 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:03:54.709200 14760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:54.728200 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.727704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45404)
0923 21:03:54.727918 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.728047 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 21:03:54.729401 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.728996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45408)
0923 21:03:54.729166 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.729251 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 21:03:54.765308 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.764707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45424)
0923 21:03:54.764959 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.765088 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260923 21:03:54.766636 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.766059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45426)
0923 21:03:54.766250 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.766390 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 21:03:54.802500 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14637
I20260923 21:03:54.819257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14276
I20260923 21:03:54.820659 14338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:54.934410 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:54.933745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45438)
0923 21:03:54.933987 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:54.934121 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 21:03:54.935232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14276
I20260923 21:03:54.957315 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44601
--webserver_interface=127.0.97.62
--webserver_port=42451
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:55.308799 14816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:55.309118 14816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:55.309281 14816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:55.317567 14816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:55.317689 14816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:55.317772 14816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:55.317876 14816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:55.317957 14816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:55.328861 14816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:55.330284 14816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:55.331887 14816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:55.357528 14822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:55.357636 14825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:55.357595 14823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:55.360262 14816 server_base.cc:1034] running on GCE node
I20260923 21:03:55.361359 14816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:55.363500 14816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:55.364739 14816 hybrid_clock.cc:648] HybridClock initialized: now 1790197435364684 us; error 36 us; skew 500 ppm
I20260923 21:03:55.365232 14816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:55 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:55.369606 14816 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:03:55.371660 14816 webserver.cc:460] Webserver started at http://127.0.97.62:42451/ using document root <none> and password file <none>
I20260923 21:03:55.372511 14816 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:55.372674 14816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:55.379760 14816 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:03:55.383940 14832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:55.385582 14816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:03:55.385751 14816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "70f55b10ad9e4388884c9e03137f2e79"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "cd4a4b485ec5c8f3fe32d06e431b810c"
server_key_iv: "32d5f6bb3be6fca0266e49b3a554c0e3"
server_key_version: "encryptionkey@0"
I20260923 21:03:55.386981 14816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:55.404379 14816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:55.409540 14816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:55.409817 14816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:55.430755 14816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44601
I20260923 21:03:55.430764 14883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44601 every 8 connection(s)
I20260923 21:03:55.432426 14816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:03:55.434314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14816
I20260923 21:03:55.435364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36223
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44601
--webserver_port=42137
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:55.443280 14884 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:03:55.447083 14884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Bootstrap starting.
I20260923 21:03:55.467911 14884 log.cc:824] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:55.480718 14884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:03:55.481374 14884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Bootstrap complete.
I20260923 21:03:55.491487 14884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:55.492563 14884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f55b10ad9e4388884c9e03137f2e79, State: Initialized, Role: FOLLOWER
I20260923 21:03:55.493221 14884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:55.493762 14884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:03:55.493975 14884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:03:55.494186 14884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 1 FOLLOWER]: Advancing to term 2
I20260923 21:03:55.498631 14884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:55.499230 14884 leader_election.cc:302] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f55b10ad9e4388884c9e03137f2e79; no voters:
I20260923 21:03:55.499957 14884 leader_election.cc:288] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 21:03:55.500106 14888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 21:03:55.501250 14888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 LEADER]: Becoming Leader. State: Replica: 70f55b10ad9e4388884c9e03137f2e79, State: Running, Role: LEADER
I20260923 21:03:55.501870 14888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:55.502452 14884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:55.504554 14890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f55b10ad9e4388884c9e03137f2e79. Latest consensus state: current_term: 2 leader_uuid: "70f55b10ad9e4388884c9e03137f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } } }
I20260923 21:03:55.504932 14890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:55.504732 14889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "70f55b10ad9e4388884c9e03137f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } } }
I20260923 21:03:55.505390 14889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:55.509069 14895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:03:55.518735 14895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:03:55.520581 14895 catalog_manager.cc:1259] Loaded cluster ID: 0a95063041be46929c73b51a26339735
I20260923 21:03:55.520761 14895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:03:55.524588 14895 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 21:03:55.544857 14895 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 21:03:55.554920 14895 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:03:55.556938 14895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Loaded TSK: 0
I20260923 21:03:55.558732 14895 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 21:03:55.813771 14886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:55.814105 14886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:55.814270 14886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:55.821975 14886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:55.822146 14886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:55.822319 14886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:03:55.832585 14886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42137
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:55.834009 14886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:55.835377 14886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:55.851501 14911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:55.851859 14912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:55.852233 14914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:55.853561 14886 server_base.cc:1034] running on GCE node
I20260923 21:03:55.854512 14886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:55.855914 14886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:55.857225 14886 hybrid_clock.cc:648] HybridClock initialized: now 1790197435857162 us; error 81 us; skew 500 ppm
I20260923 21:03:55.857657 14886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:55 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:55.862268 14886 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:03:55.864275 14886 webserver.cc:460] Webserver started at http://127.0.97.1:42137/ using document root <none> and password file <none>
I20260923 21:03:55.865257 14886 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:55.865425 14886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:55.873562 14886 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:03:55.877564 14921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:55.879443 14886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:03:55.879655 14886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "0abee61659ed40309cf450b131da4867"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "891b11d444f56756ddda660fbb17286e"
server_key_iv: "fea98379e9fbfbf31a76acd418944c82"
server_key_version: "encryptionkey@0"
I20260923 21:03:55.880795 14886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:55.905674 14886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:55.910461 14886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:55.910789 14886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:55.911882 14886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:55.913337 14886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:55.913429 14886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:55.913595 14886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:55.913677 14886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:55.950277 14886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36223
I20260923 21:03:55.950460 15033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36223 every 8 connection(s)
I20260923 21:03:55.951867 14886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:03:55.953177 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 14886
I20260923 21:03:55.954007 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46123
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44601
--webserver_port=42131
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:03:55 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:55.979892 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:55.955233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52763 (local address 127.0.97.62:44601)
0923 21:03:55.955871 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:55.955883 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:55.956668 (+ 785us) server_negotiation.cc:407] Connection header received
0923 21:03:55.958965 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:55.958981 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:55.959345 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:55.959740 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:55.961374 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:55.963211 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:55.966120 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:55.966892 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:55.967146 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:55.971416 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:55.971426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:55.971445 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:55.971490 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:55.974650 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:55.975822 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:55.975829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:55.975835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:55.976017 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:55.976618 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:55.976622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:55.976624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:55.977147 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:55.977292 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:55.978480 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0923 21:03:55.978869 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":292,"threads_started":1}
I20260923 21:03:55.980340 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:55.955668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:52763)
0923 21:03:55.956465 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:55.956514 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:55.958300 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:55.959909 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:55.959921 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:55.960586 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:55.961130 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:55.961149 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:55.963493 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:55.963499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:55.965938 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:55.965956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:55.967188 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:55.967195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:55.967509 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:55.968240 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:55.968264 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:55.971045 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:55.974842 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:55.974857 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:55.974877 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:55.975665 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:55.976175 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:55.976181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:55.976185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:55.976490 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:55.977314 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:55.977325 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:55.977862 (+ 537us) client_negotiation.cc:769] Sending connection context
0923 21:03:55.978241 (+ 379us) client_negotiation.cc:240] Negotiation successful
0923 21:03:55.978950 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":347,"threads_started":1}
I20260923 21:03:55.983618 15034 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44601
I20260923 21:03:55.983975 15034 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:55.984920 15034 heartbeater.cc:511] Master 127.0.97.62:44601 requested a full tablet report, sending...
I20260923 21:03:55.987551 14849 ts_manager.cc:194] Registered new tserver with Master: 0abee61659ed40309cf450b131da4867 (127.0.97.1:36223)
I20260923 21:03:55.989188 14849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52763
W20260923 21:03:56.310101 15038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:56.310668 15038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:56.310887 15038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:56.318578 15038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:56.318765 15038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:56.318961 15038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:03:56.329273 15038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42131
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:56.330783 15038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:56.332222 15038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:56.346727 15043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:56.346766 15044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:56.347038 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:56.348121 15038 server_base.cc:1034] running on GCE node
I20260923 21:03:56.348773 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:56.349948 15038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:56.351186 15038 hybrid_clock.cc:648] HybridClock initialized: now 1790197436351136 us; error 45 us; skew 500 ppm
I20260923 21:03:56.351599 15038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:56 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:56.355062 15038 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:03:56.356750 15038 webserver.cc:460] Webserver started at http://127.0.97.2:42131/ using document root <none> and password file <none>
I20260923 21:03:56.357482 15038 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:56.357647 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:56.363590 15038 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:03:56.366796 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:56.368481 15038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 21:03:56.368680 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "db64176508ee4dae92ba5d20f0dd7fd1"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "5c25dd94422135b8302409adf252635a"
server_key_iv: "81c5cdfca3c911d1e42da2962376da7b"
server_key_version: "encryptionkey@0"
I20260923 21:03:56.369763 15038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:56.383512 15038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:56.388639 15038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:56.388938 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:56.389994 15038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:56.391780 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:56.391899 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:56.392041 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:56.392135 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:56.429771 15038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46123
I20260923 21:03:56.429857 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46123 every 8 connection(s)
I20260923 21:03:56.431524 15038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:03:56.441112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 15038
I20260923 21:03:56.441869 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43187
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44601
--webserver_port=34123
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:03:56 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:56.455368 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:56.434746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34701 (local address 127.0.97.62:44601)
0923 21:03:56.434956 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:56.434961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:56.436174 (+ 1213us) server_negotiation.cc:407] Connection header received
0923 21:03:56.438179 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:56.438185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:56.438387 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:56.438521 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:56.440198 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.440970 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.442954 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.443507 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.443660 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:56.447998 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:56.448004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:56.448007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:56.448047 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:56.451775 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:56.452664 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:56.452670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:56.452672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:56.452818 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:56.453325 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:56.453329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:56.453331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:56.453650 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:56.453830 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:56.454699 (+ 869us) server_negotiation.cc:299] Negotiation successful
0923 21:03:56.454971 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 21:03:56.456477 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:56.435140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:34701)
0923 21:03:56.435971 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:56.436013 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:56.437881 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:56.438802 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:56.438812 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:56.439383 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:56.440007 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:56.440026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.441148 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.441159 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:56.442787 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:56.442802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.443794 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.443804 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:56.444056 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:56.444860 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:56.444886 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:56.447707 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:56.451965 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:56.451976 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:56.452000 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:56.452496 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:56.452948 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:56.452953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:56.452957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:56.453175 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:56.453831 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:56.453843 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:56.454212 (+ 369us) client_negotiation.cc:769] Sending connection context
0923 21:03:56.454581 (+ 369us) client_negotiation.cc:240] Negotiation successful
0923 21:03:56.455178 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":316,"threads_started":1}
I20260923 21:03:56.458530 15166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44601
I20260923 21:03:56.458889 15166 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:56.459672 15166 heartbeater.cc:511] Master 127.0.97.62:44601 requested a full tablet report, sending...
I20260923 21:03:56.461440 14849 ts_manager.cc:194] Registered new tserver with Master: db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2:46123)
I20260923 21:03:56.462869 14849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34701
W20260923 21:03:56.789921 15169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:56.790232 15169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:56.790372 15169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:56.798182 15169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:56.798305 15169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:56.798539 15169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:03:56.808858 15169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34123
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:56.810273 15169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:56.811653 15169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:56.825954 15175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:56.826085 15174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:56.825974 15177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:56.832181 15169 server_base.cc:1034] running on GCE node
I20260923 21:03:56.832945 15169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:56.834028 15169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:56.835256 15169 hybrid_clock.cc:648] HybridClock initialized: now 1790197436835195 us; error 57 us; skew 500 ppm
I20260923 21:03:56.835601 15169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:56 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:56.839041 15169 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:03:56.840731 15169 webserver.cc:460] Webserver started at http://127.0.97.3:34123/ using document root <none> and password file <none>
I20260923 21:03:56.841486 15169 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:56.841641 15169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:56.847618 15169 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:03:56.850710 15184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:56.852210 15169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 21:03:56.852402 15169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "9aceb9d6306048389443d0e34a05848b"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "0e379526ebaf6da5d07c236123b016ef"
server_key_iv: "bbc145666c708ab631e8c542c4b98973"
server_key_version: "encryptionkey@0"
I20260923 21:03:56.853449 15169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:56.903121 15169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:56.907905 15169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:56.908211 15169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:56.909523 15169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:56.911128 15169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:56.911214 15169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:56.911352 15169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:56.911441 15169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:56.946115 15169 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43187
I20260923 21:03:56.946209 15296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43187 every 8 connection(s)
I20260923 21:03:56.947660 15169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:03:56.950230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 15169
Sep 23 21:03:56 dist-test-slave-8xn5 krb5kdc[14263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:03:56.977630 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:56.951171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.0.97.62:44601)
0923 21:03:56.960727 (+ 9556us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:56.960731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:56.960744 (+ 13us) server_negotiation.cc:407] Connection header received
0923 21:03:56.960795 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:56.960798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:56.960864 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:56.961005 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:56.962794 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.963682 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.965215 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.965656 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.965795 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:56.969773 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:56.969781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:56.969786 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:56.969825 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:56.974469 (+ 4644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:56.975381 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:56.975389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:56.975393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:56.975563 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:56.975944 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:56.975949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:56.975954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:56.976373 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:56.976562 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:56.977002 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 21:03:56.977231 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":9462,"thread_start_us":9347,"threads_started":1}
I20260923 21:03:56.980223 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:56.950672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58449 (local address 127.0.97.62:44601)
0923 21:03:56.950877 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:56.950887 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:56.960499 (+ 9612us) server_negotiation.cc:407] Connection header received
0923 21:03:56.962506 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:56.962512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:56.962597 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:56.962722 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:56.964330 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.965062 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.967200 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.967868 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.968014 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:56.972695 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:56.972705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:56.972711 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:56.972765 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:56.976188 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:56.977198 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:56.977205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:56.977208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:56.977328 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:56.977972 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:56.977980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:56.977985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:56.978408 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:56.978542 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:56.979620 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0923 21:03:56.979847 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":76}
I20260923 21:03:56.981258 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:56.951317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:58449)
0923 21:03:56.960237 (+ 8920us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:56.960320 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:56.962166 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:56.963005 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:56.963016 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:56.963566 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:03:56.964153 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:56.964171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.965230 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.965234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:56.966988 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:56.967006 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.968334 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.968344 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:56.968650 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:56.969666 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:03:56.969699 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:03:56.972428 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:03:56.976340 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:56.976348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:56.976363 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:56.977067 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:56.977584 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:03:56.977588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:03:56.977591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:03:56.977829 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:03:56.978622 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:03:56.978631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:03:56.979136 (+ 505us) client_negotiation.cc:769] Sending connection context
0923 21:03:56.979459 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 21:03:56.979890 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8427,"thread_start_us":8107,"threads_started":1}
I20260923 21:03:56.982802 15297 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44601
I20260923 21:03:56.983197 15297 heartbeater.cc:464] Registering TS with master...
I20260923 21:03:56.984004 15297 heartbeater.cc:511] Master 127.0.97.62:44601 requested a full tablet report, sending...
I20260923 21:03:56.985574 14849 ts_manager.cc:194] Registered new tserver with Master: 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187)
I20260923 21:03:56.986552 14849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58449
I20260923 21:03:56.992920 15034 heartbeater.cc:503] Master 127.0.97.62:44601 was elected leader, sending a full tablet report...
I20260923 21:03:56.993269 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:03:57.000990 15036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:56.994390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.0.97.62:44601)
0923 21:03:56.994583 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:56.994589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:56.995124 (+ 535us) server_negotiation.cc:407] Connection header received
0923 21:03:56.995303 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:56.995306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:56.995523 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:56.995655 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:03:56.996143 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.997126 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.998751 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:56.999178 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:56.999275 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:56.999831 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:03:57.000376 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:03:57.000490 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.000598 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.000686 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 21:03:57.001971 14849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:03:57.009743 14849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:03:57.032347 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.022776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.0.97.1:36223)
0923 21:03:57.023511 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.023520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.023764 (+ 244us) server_negotiation.cc:407] Connection header received
0923 21:03:57.024024 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.024033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.024315 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.024498 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:57.025567 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.027131 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.029524 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.031065 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.031182 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.031368 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.031829 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.032006 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":760,"threads_started":1}
I20260923 21:03:57.032797 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.022503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.0.1:45464)
0923 21:03:57.023461 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:57.023503 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:57.023685 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:57.024819 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:57.024826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:57.024862 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:57.025305 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.025317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.027264 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.027272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.029314 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.029328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.031474 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.031479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.031599 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.031633 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 21:03:57.031771 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 21:03:57.031972 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":71,"thread_start_us":1177,"threads_started":1}
I20260923 21:03:57.033347 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.023273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43187 (local address 127.0.0.1:36284)
0923 21:03:57.024214 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:57.024232 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:57.024360 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:57.025638 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:57.025644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:57.025671 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:57.026011 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.026022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.028318 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.028323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.030649 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.030663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.032479 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.032486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.032651 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.032693 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 21:03:57.032754 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 21:03:57.032896 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":25,"thread_start_us":841,"threads_started":1}
I20260923 21:03:57.033787 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.024457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46123 (local address 127.0.0.1:59470)
0923 21:03:57.025305 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:57.025322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:57.025437 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:57.026554 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:57.026558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:57.026583 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:57.026983 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.026996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.028794 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.028802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.030984 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.030999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.032867 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.032873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.033107 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.033150 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 21:03:57.033239 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 21:03:57.033368 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":3,"thread_start_us":677,"threads_started":1}
I20260923 21:03:57.034649 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.023893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.0.97.3:43187)
0923 21:03:57.025017 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.025026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.025061 (+ 35us) server_negotiation.cc:407] Connection header received
0923 21:03:57.025163 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.025172 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.025430 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.026213 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:57.026549 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.028158 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.030810 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.032198 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.033228 (+ 1030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.033786 (+ 558us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.034027 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.034212 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":787,"threads_started":1}
I20260923 21:03:57.034914 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.024973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.97.2:46123)
0923 21:03:57.025924 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.025933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.025963 (+ 30us) server_negotiation.cc:407] Connection header received
0923 21:03:57.026060 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.026067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.026301 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.027164 (+ 863us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:57.027399 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.028663 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.031159 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.032590 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.032727 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.032922 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.033592 (+ 670us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.034295 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":690,"threads_started":1}
I20260923 21:03:57.037205 14969 tablet_service.cc:1467] Processing CreateTablet for tablet ad2986597de5442e8ed7ed08270fa3ad (DEFAULT_TABLE table=test-table [id=7ef748e401ed4d64b515daeeb08b3266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:57.038532 14969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2986597de5442e8ed7ed08270fa3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:57.038277 15232 tablet_service.cc:1467] Processing CreateTablet for tablet ad2986597de5442e8ed7ed08270fa3ad (DEFAULT_TABLE table=test-table [id=7ef748e401ed4d64b515daeeb08b3266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:57.038278 15101 tablet_service.cc:1467] Processing CreateTablet for tablet ad2986597de5442e8ed7ed08270fa3ad (DEFAULT_TABLE table=test-table [id=7ef748e401ed4d64b515daeeb08b3266]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:03:57.039686 15232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2986597de5442e8ed7ed08270fa3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:57.039686 15101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2986597de5442e8ed7ed08270fa3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:03:57.051257 15313 tablet_bootstrap.cc:492] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Bootstrap starting.
I20260923 21:03:57.055614 15315 tablet_bootstrap.cc:492] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Bootstrap starting.
I20260923 21:03:57.055929 15313 tablet_bootstrap.cc:654] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:57.057826 15314 tablet_bootstrap.cc:492] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Bootstrap starting.
I20260923 21:03:57.057854 15313 log.cc:824] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:57.060333 15315 tablet_bootstrap.cc:654] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:57.061409 15315 log.cc:824] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:57.061861 15313 tablet_bootstrap.cc:492] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: No bootstrap required, opened a new log
I20260923 21:03:57.062454 15313 ts_tablet_manager.cc:1397] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 21:03:57.062664 15314 tablet_bootstrap.cc:654] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Neither blocks nor log segments found. Creating new log.
I20260923 21:03:57.063979 15314 log.cc:824] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:57.065019 15315 tablet_bootstrap.cc:492] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: No bootstrap required, opened a new log
I20260923 21:03:57.065303 15315 ts_tablet_manager.cc:1397] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260923 21:03:57.067924 15314 tablet_bootstrap.cc:492] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: No bootstrap required, opened a new log
I20260923 21:03:57.068267 15314 ts_tablet_manager.cc:1397] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 21:03:57.072942 15315 raft_consensus.cc:348] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.073458 15315 raft_consensus.cc:374] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:57.073557 15315 raft_consensus.cc:729] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db64176508ee4dae92ba5d20f0dd7fd1, State: Initialized, Role: FOLLOWER
I20260923 21:03:57.074395 15315 consensus_queue.cc:260] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.075345 15313 raft_consensus.cc:348] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.075810 15166 heartbeater.cc:503] Master 127.0.97.62:44601 was elected leader, sending a full tablet report...
I20260923 21:03:57.075898 15313 raft_consensus.cc:374] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:57.076026 15313 raft_consensus.cc:729] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0abee61659ed40309cf450b131da4867, State: Initialized, Role: FOLLOWER
I20260923 21:03:57.076470 15315 ts_tablet_manager.cc:1428] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260923 21:03:57.076879 15313 consensus_queue.cc:260] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.080684 15313 ts_tablet_manager.cc:1428] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260923 21:03:57.081771 15314 raft_consensus.cc:348] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.082480 15314 raft_consensus.cc:374] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:03:57.082629 15314 raft_consensus.cc:729] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aceb9d6306048389443d0e34a05848b, State: Initialized, Role: FOLLOWER
I20260923 21:03:57.083506 15314 consensus_queue.cc:260] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.084923 15297 heartbeater.cc:503] Master 127.0.97.62:44601 was elected leader, sending a full tablet report...
I20260923 21:03:57.085815 15314 ts_tablet_manager.cc:1428] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260923 21:03:57.184028 15167 tablet.cc:2369] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:03:57.199352 15298 tablet.cc:2369] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:03:57.204653 15035 tablet.cc:2369] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:03:57.287858 15320 raft_consensus.cc:482] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:03:57.288213 15320 raft_consensus.cc:504] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.290277 15320 leader_election.cc:288] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187), db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2:46123)
I20260923 21:03:57.292287 15321 raft_consensus.cc:482] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:03:57.292652 15321 raft_consensus.cc:504] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.300390 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.290777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36539 (local address 127.0.97.3:43187)
0923 21:03:57.291013 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.291022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.291897 (+ 875us) server_negotiation.cc:407] Connection header received
0923 21:03:57.292149 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.292157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.292240 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.292351 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:57.293186 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.294512 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.297684 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.299130 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.299227 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.299448 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.299932 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.300087 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":85}
I20260923 21:03:57.300463 15321 leader_election.cc:288] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0abee61659ed40309cf450b131da4867 (127.0.97.1:36223), db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2:46123)
I20260923 21:03:57.301164 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.290765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43187 (local address 127.0.97.1:36539)
0923 21:03:57.291772 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:57.291793 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:57.291992 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:57.292376 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:57.292381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:57.292405 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:57.292912 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.292923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.294667 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.294672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.297468 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.297489 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.299567 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.299573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.299742 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.299775 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 21:03:57.300591 (+ 816us) client_negotiation.cc:240] Negotiation successful
0923 21:03:57.300740 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":9,"thread_start_us":670,"threads_started":1}
I20260923 21:03:57.302275 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.290778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46123 (local address 127.0.97.1:34429)
0923 21:03:57.291874 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:57.291893 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:57.292103 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:57.292555 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:57.292560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:57.292599 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:57.293065 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.293077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.294642 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.294649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.298982 (+ 4333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.298998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.301057 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.301062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.301557 (+ 495us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.301582 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 21:03:57.301690 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 21:03:57.301832 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":193,"thread_start_us":530,"threads_started":1}
I20260923 21:03:57.303015 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.290874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34429 (local address 127.0.97.2:46123)
0923 21:03:57.291067 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.291074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.291993 (+ 919us) server_negotiation.cc:407] Connection header received
0923 21:03:57.292290 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.292295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.292413 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.292510 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:57.293320 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.294495 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.299177 (+ 4682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.300397 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.300783 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.300900 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.302603 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.302752 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 21:03:57.304030 15121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2986597de5442e8ed7ed08270fa3ad" candidate_uuid: "0abee61659ed40309cf450b131da4867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" is_pre_election: true
I20260923 21:03:57.304299 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.295387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53421 (local address 127.0.97.1:36223)
0923 21:03:57.295594 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.295600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.295616 (+ 16us) server_negotiation.cc:407] Connection header received
0923 21:03:57.295672 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.295677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.295751 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.295838 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:57.297248 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.298628 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.301706 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.302986 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.303128 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.303233 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.303931 (+ 698us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.304064 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46}
I20260923 21:03:57.304654 15121 raft_consensus.cc:2454] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0abee61659ed40309cf450b131da4867 in term 0.
I20260923 21:03:57.304740 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.294950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36223 (local address 127.0.97.3:53421)
0923 21:03:57.295155 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:57.295176 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:57.295287 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:57.296758 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:57.296763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:57.296791 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:57.297072 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.297082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.298803 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.298812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.301495 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.301517 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.303304 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.303314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.303663 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.303779 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 21:03:57.303911 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 21:03:57.304111 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 21:03:57.305357 14925 leader_election.cc:302] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0abee61659ed40309cf450b131da4867, db64176508ee4dae92ba5d20f0dd7fd1; no voters:
I20260923 21:03:57.305792 15320 raft_consensus.cc:2781] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:03:57.305953 15320 raft_consensus.cc:482] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:03:57.306062 15320 raft_consensus.cc:3037] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:57.307561 15252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2986597de5442e8ed7ed08270fa3ad" candidate_uuid: "0abee61659ed40309cf450b131da4867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aceb9d6306048389443d0e34a05848b" is_pre_election: true
I20260923 21:03:57.308102 15252 raft_consensus.cc:2454] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0abee61659ed40309cf450b131da4867 in term 0.
I20260923 21:03:57.308846 14989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2986597de5442e8ed7ed08270fa3ad" candidate_uuid: "9aceb9d6306048389443d0e34a05848b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0abee61659ed40309cf450b131da4867" is_pre_election: true
I20260923 21:03:57.309620 15320 raft_consensus.cc:504] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.310191 14989 raft_consensus.cc:2379] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9aceb9d6306048389443d0e34a05848b in current term 1: Already voted for candidate 0abee61659ed40309cf450b131da4867 in this term.
I20260923 21:03:57.310607 15320 leader_election.cc:288] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [CANDIDATE]: Term 1 election: Requested vote from peers 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187), db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2:46123)
I20260923 21:03:57.311189 15252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2986597de5442e8ed7ed08270fa3ad" candidate_uuid: "0abee61659ed40309cf450b131da4867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aceb9d6306048389443d0e34a05848b"
I20260923 21:03:57.311318 15121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2986597de5442e8ed7ed08270fa3ad" candidate_uuid: "0abee61659ed40309cf450b131da4867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db64176508ee4dae92ba5d20f0dd7fd1"
I20260923 21:03:57.311504 15252 raft_consensus.cc:3037] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:57.311555 15121 raft_consensus.cc:3037] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:03:57.313135 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.295845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46123 (local address 127.0.97.3:54561)
0923 21:03:57.305030 (+ 9185us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:57.305055 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 21:03:57.305196 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:03:57.305655 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:03:57.305662 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:03:57.305691 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:03:57.305973 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.305988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.307693 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.307700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.309814 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:03:57.309829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.312230 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.312237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:03:57.312481 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.312522 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 21:03:57.312633 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 21:03:57.312794 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9043,"thread_start_us":10688,"threads_started":1}
I20260923 21:03:57.313191 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.296410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54561 (local address 127.0.97.2:46123)
0923 21:03:57.298075 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.298083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.305327 (+ 7244us) server_negotiation.cc:407] Connection header received
0923 21:03:57.305409 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.305417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.305518 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.305635 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:03:57.306226 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.307544 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.310430 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.312036 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.312177 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.312321 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.312675 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.312832 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1473,"thread_start_us":2276,"threads_started":1}
I20260923 21:03:57.314174 15120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2986597de5442e8ed7ed08270fa3ad" candidate_uuid: "9aceb9d6306048389443d0e34a05848b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" is_pre_election: true
I20260923 21:03:57.314787 15121 raft_consensus.cc:2454] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0abee61659ed40309cf450b131da4867 in term 1.
I20260923 21:03:57.314793 15252 raft_consensus.cc:2454] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0abee61659ed40309cf450b131da4867 in term 1.
I20260923 21:03:57.315186 14922 leader_election.cc:302] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0abee61659ed40309cf450b131da4867, 9aceb9d6306048389443d0e34a05848b; no voters:
I20260923 21:03:57.315636 15320 raft_consensus.cc:2781] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:03:57.315735 15188 leader_election.cc:302] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9aceb9d6306048389443d0e34a05848b; no voters: 0abee61659ed40309cf450b131da4867, db64176508ee4dae92ba5d20f0dd7fd1
I20260923 21:03:57.316239 15321 raft_consensus.cc:2726] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 21:03:57.316603 15320 raft_consensus.cc:686] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 1 LEADER]: Becoming Leader. State: Replica: 0abee61659ed40309cf450b131da4867, State: Running, Role: LEADER
I20260923 21:03:57.317171 15320 consensus_queue.cc:237] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } }
I20260923 21:03:57.322101 14848 catalog_manager.cc:5495] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0abee61659ed40309cf450b131da4867 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0abee61659ed40309cf450b131da4867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0abee61659ed40309cf450b131da4867" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 } health_report { overall_health: UNKNOWN } } }
I20260923 21:03:57.404170 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.397947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.0.97.1:36223)
0923 21:03:57.398231 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.398240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.398262 (+ 22us) server_negotiation.cc:407] Connection header received
0923 21:03:57.398562 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.398567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.398664 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.398841 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:03:57.399413 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.400231 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.401753 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.402194 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.402336 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.402973 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:03:57.403501 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:03:57.403673 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.403732 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.403847 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 21:03:57.413687 15121 raft_consensus.cc:1261] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 FOLLOWER]: Refusing update from remote peer 0abee61659ed40309cf450b131da4867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:57.413744 15252 raft_consensus.cc:1261] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Refusing update from remote peer 0abee61659ed40309cf450b131da4867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:03:57.414883 15326 consensus_queue.cc:1035] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db64176508ee4dae92ba5d20f0dd7fd1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:57.415211 15320 consensus_queue.cc:1035] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aceb9d6306048389443d0e34a05848b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:03:57.427692 15329 mvcc.cc:205] Tried to move back new op lower bound from 7332648703636480000 to 7332648703255650304. Current Snapshot: MvccSnapshot[applied={T|T < 7332648703636480000}]
I20260923 21:03:57.453138 14848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:03:57.453519 14848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:03:57.458153 14848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Sending DeleteTablet for 3 replicas of tablet ad2986597de5442e8ed7ed08270fa3ad
I20260923 21:03:57.459764 14969 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2986597de5442e8ed7ed08270fa3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45464
I20260923 21:03:57.460595 15232 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2986597de5442e8ed7ed08270fa3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36284
I20260923 21:03:57.461537 15339 tablet_replica.cc:330] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: stopping tablet replica
I20260923 21:03:57.462006 15339 raft_consensus.cc:2229] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:03:57.462263 15101 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2986597de5442e8ed7ed08270fa3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:03:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59470
I20260923 21:03:57.462726 15339 raft_consensus.cc:2258] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:57.464789 15339 ts_tablet_manager.cc:1905] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:57.469990 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14886
I20260923 21:03:57.471060 15340 tablet_replica.cc:330] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: stopping tablet replica
I20260923 21:03:57.471522 15340 raft_consensus.cc:2229] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:03:57.472052 15340 raft_consensus.cc:2258] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:57.472872 15338 tablet_replica.cc:330] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: stopping tablet replica
I20260923 21:03:57.473145 15339 ts_tablet_manager.cc:1918] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:57.473363 15339 log.cc:1196] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/ad2986597de5442e8ed7ed08270fa3ad
I20260923 21:03:57.473417 15338 raft_consensus.cc:2229] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:03:57.474063 15339 ts_tablet_manager.cc:1939] T ad2986597de5442e8ed7ed08270fa3ad P 9aceb9d6306048389443d0e34a05848b: Deleting consensus metadata
I20260923 21:03:57.474186 15338 raft_consensus.cc:2258] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:03:57.476534 15340 ts_tablet_manager.cc:1905] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:57.477793 15338 ts_tablet_manager.cc:1905] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:03:57.478878 14833 catalog_manager.cc:4841] TS 9aceb9d6306048389443d0e34a05848b (127.0.97.3:43187): tablet ad2986597de5442e8ed7ed08270fa3ad (table test-table [id=7ef748e401ed4d64b515daeeb08b3266]) successfully deleted
I20260923 21:03:57.483999 15340 ts_tablet_manager.cc:1918] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:57.484201 15340 log.cc:1196] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/ad2986597de5442e8ed7ed08270fa3ad
I20260923 21:03:57.484781 15340 ts_tablet_manager.cc:1939] T ad2986597de5442e8ed7ed08270fa3ad P db64176508ee4dae92ba5d20f0dd7fd1: Deleting consensus metadata
I20260923 21:03:57.485662 15338 ts_tablet_manager.cc:1918] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:03:57.485841 15338 log.cc:1196] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/ad2986597de5442e8ed7ed08270fa3ad
I20260923 21:03:57.486514 15338 ts_tablet_manager.cc:1939] T ad2986597de5442e8ed7ed08270fa3ad P 0abee61659ed40309cf450b131da4867: Deleting consensus metadata
I20260923 21:03:57.486971 14836 catalog_manager.cc:4841] TS db64176508ee4dae92ba5d20f0dd7fd1 (127.0.97.2:46123): tablet ad2986597de5442e8ed7ed08270fa3ad (table test-table [id=7ef748e401ed4d64b515daeeb08b3266]) successfully deleted
I20260923 21:03:57.488065 14836 catalog_manager.cc:4841] TS 0abee61659ed40309cf450b131da4867 (127.0.97.1:36223): tablet ad2986597de5442e8ed7ed08270fa3ad (table test-table [id=7ef748e401ed4d64b515daeeb08b3266]) successfully deleted
I20260923 21:03:57.516700 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.472028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.0.97.1:36223)
0923 21:03:57.472311 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.472321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.472347 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:03:57.472431 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.472441 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.472545 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.472686 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:57.474891 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.476182 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.479287 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.480172 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.480320 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.509589 (+ 29269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:57.509613 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:57.509635 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:57.509707 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:57.513525 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:57.514424 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:57.514432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:57.514434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:57.514573 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:57.515022 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:57.515026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:57.515028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:57.515436 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:57.515798 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.516098 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.516336 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 21:03:57.517374 15029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:57.623054 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14886
I20260923 21:03:57.642300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 15038
I20260923 21:03:57.658987 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.643220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.97.2:46123)
0923 21:03:57.643432 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.643440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.643641 (+ 201us) server_negotiation.cc:407] Connection header received
0923 21:03:57.643891 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.643896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.643977 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.644097 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:57.645616 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.646482 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.648451 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.648871 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.648986 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.652161 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:57.652177 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:57.652193 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:57.652231 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:57.655549 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:57.656412 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:57.656420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:57.656422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:57.656571 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:57.657153 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:57.657164 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:57.657166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:57.657630 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:57.657978 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.658490 (+ 512us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.658730 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 21:03:57.659832 15161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:57.752017 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 15038
I20260923 21:03:57.770457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 15169
I20260923 21:03:57.785579 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:57.771315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.0.97.3:43187)
0923 21:03:57.771537 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 21:03:57.771542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:03:57.771633 (+ 91us) server_negotiation.cc:407] Connection header received
0923 21:03:57.771782 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:03:57.771787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:03:57.771867 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:03:57.771955 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:03:57.773488 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.774239 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.775893 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:03:57.776405 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:03:57.776524 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:03:57.779235 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:03:57.779250 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:03:57.779268 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:03:57.779309 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:03:57.782651 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:57.783479 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:57.783488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:57.783490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:57.783651 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:03:57.784033 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:03:57.784037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:03:57.784039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:03:57.784523 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:03:57.784755 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 21:03:57.785114 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 21:03:57.785292 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260923 21:03:57.786293 15292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:57.878551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 15169
I20260923 21:03:57.896234 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 14816
I20260923 21:03:57.897557 14879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:03:58.022187 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 14816
I20260923 21:03:58.045409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44601
--webserver_interface=127.0.97.62
--webserver_port=42451
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:03:58.397387 15345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:58.397699 15345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:58.397871 15345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:58.405526 15345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:03:58.405659 15345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:58.405740 15345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:58.405826 15345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:03:58.405947 15345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:03:58.416370 15345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:58.417891 15345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:58.419373 15345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:58.432278 15351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:58.432459 15350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:58.432375 15353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:58.433332 15345 server_base.cc:1034] running on GCE node
I20260923 21:03:58.434224 15345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:58.435907 15345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:58.437176 15345 hybrid_clock.cc:648] HybridClock initialized: now 1790197438437060 us; error 120 us; skew 500 ppm
I20260923 21:03:58.437620 15345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:58 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:58.441272 15345 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:03:58.443023 15345 webserver.cc:460] Webserver started at http://127.0.97.62:42451/ using document root <none> and password file <none>
I20260923 21:03:58.443787 15345 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:58.443941 15345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:58.449949 15345 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 21:03:58.453215 15360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:58.454968 15345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 21:03:58.455168 15345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "70f55b10ad9e4388884c9e03137f2e79"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "cd4a4b485ec5c8f3fe32d06e431b810c"
server_key_iv: "32d5f6bb3be6fca0266e49b3a554c0e3"
server_key_version: "encryptionkey@0"
I20260923 21:03:58.456225 15345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:58.479146 15345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:58.484612 15345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:58.484953 15345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:58.505420 15345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44601
I20260923 21:03:58.505481 15411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44601 every 8 connection(s)
I20260923 21:03:58.507367 15345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:03:58.509931 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 15345
I20260923 21:03:58.510866 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:36223
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44601
--webserver_port=42137
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:58.514894 15412 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:03:58.518790 15412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Bootstrap starting.
I20260923 21:03:58.550763 15412 log.cc:824] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Log is configured to *not* fsync() on all Append() calls
I20260923 21:03:58.572707 15412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:03:58.573352 15412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79: Bootstrap complete.
I20260923 21:03:58.584096 15412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:58.585157 15412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f55b10ad9e4388884c9e03137f2e79, State: Initialized, Role: FOLLOWER
I20260923 21:03:58.585917 15412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:58.586195 15412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:03:58.586364 15412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:03:58.586572 15412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 2 FOLLOWER]: Advancing to term 3
I20260923 21:03:58.590934 15412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:58.591529 15412 leader_election.cc:302] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70f55b10ad9e4388884c9e03137f2e79; no voters:
I20260923 21:03:58.592274 15412 leader_election.cc:288] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 21:03:58.592482 15416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 21:03:58.593842 15416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [term 3 LEADER]: Becoming Leader. State: Replica: 70f55b10ad9e4388884c9e03137f2e79, State: Running, Role: LEADER
I20260923 21:03:58.594837 15412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:03:58.594743 15416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } }
I20260923 21:03:58.599810 15417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "70f55b10ad9e4388884c9e03137f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } } }
I20260923 21:03:58.600224 15417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:58.601011 15418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70f55b10ad9e4388884c9e03137f2e79. Latest consensus state: current_term: 3 leader_uuid: "70f55b10ad9e4388884c9e03137f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70f55b10ad9e4388884c9e03137f2e79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44601 } } }
I20260923 21:03:58.601888 15418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70f55b10ad9e4388884c9e03137f2e79 [sys.catalog]: This master's current role is: LEADER
I20260923 21:03:58.607986 15422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:03:58.621004 15422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:03:58.623756 15422 catalog_manager.cc:1259] Loaded cluster ID: 0a95063041be46929c73b51a26339735
I20260923 21:03:58.624038 15422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 21:03:58.627106 15422 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb1f80079ad google::LogMessage::Fail() at ??:0
@ 0x7fb1f800a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb1f80074de google::LogMessage::Flush() at ??:0
@ 0x7fb1f8008a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb20da3a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb20da37976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb20dabf610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb20dabf439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1fac53345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb1fac5691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb1fac56799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb1fac31eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb1fdf0b6db start_thread at ??:0
@ 0x7fb1f6ea161f clone at ??:0
W20260923 21:03:58.974684 15414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:58.975111 15414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:58.975296 15414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:58.983775 15414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:58.983958 15414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:58.984158 15414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:03:58.994935 15414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:36223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42137
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:58.996479 15414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:58.997880 15414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:59.012074 15454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:59.012142 15455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:59.012378 15457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:59.013666 15414 server_base.cc:1034] running on GCE node
I20260923 21:03:59.014456 15414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:59.015479 15414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:59.016703 15414 hybrid_clock.cc:648] HybridClock initialized: now 1790197439016630 us; error 62 us; skew 500 ppm
I20260923 21:03:59.017091 15414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:59 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:59.020751 15414 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:03:59.022547 15414 webserver.cc:460] Webserver started at http://127.0.97.1:42137/ using document root <none> and password file <none>
I20260923 21:03:59.023326 15414 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:59.023492 15414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:59.029235 15414 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 21:03:59.032244 15464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:59.033906 15414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:03:59.034129 15414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "0abee61659ed40309cf450b131da4867"
format_stamp: "Formatted at 2026-09-23 21:03:52 on dist-test-slave-8xn5"
server_key: "891b11d444f56756ddda660fbb17286e"
server_key_iv: "fea98379e9fbfbf31a76acd418944c82"
server_key_version: "encryptionkey@0"
I20260923 21:03:59.035377 15414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:59.053510 15414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:59.058516 15414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:59.058816 15414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:59.059857 15414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:59.061393 15414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:59.061527 15414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:59.061657 15414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:59.061750 15414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:59.097927 15414 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36223
I20260923 21:03:59.098017 15576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36223 every 8 connection(s)
I20260923 21:03:59.099502 15414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:03:59.102476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 15414
I20260923 21:03:59.103307 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46123
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44601
--webserver_port=42131
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:59.105284 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:59.103085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:47259)
0923 21:03:59.104062 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:59.104335 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":465,"threads_started":1}
W20260923 21:03:59.106128 15577 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44601 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44601: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
I20260923 21:03:59.107125 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:59.106772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:59087)
0923 21:03:59.106938 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:59.107018 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 21:03:59.108135 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:59.107800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:44213)
0923 21:03:59.107945 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:59.108028 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 21:03:59.108491 15577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 21:03:59.468083 15580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:59.468379 15580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:59.468541 15580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:59.476210 15580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:59.476341 15580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:59.476522 15580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:03:59.487107 15580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42131
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:59.488600 15580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:59.490044 15580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:59.504544 15585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:59.504525 15586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:59.504750 15588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:59.505774 15580 server_base.cc:1034] running on GCE node
I20260923 21:03:59.506667 15580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:59.507745 15580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:59.509020 15580 hybrid_clock.cc:648] HybridClock initialized: now 1790197439508951 us; error 68 us; skew 500 ppm
I20260923 21:03:59.509510 15580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:59 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:03:59.513127 15580 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:03:59.514896 15580 webserver.cc:460] Webserver started at http://127.0.97.2:42131/ using document root <none> and password file <none>
I20260923 21:03:59.515663 15580 fs_manager.cc:359] Metadata directory not provided
I20260923 21:03:59.515828 15580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:03:59.521948 15580 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 21:03:59.525125 15595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:59.526674 15580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 21:03:59.526871 15580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "db64176508ee4dae92ba5d20f0dd7fd1"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "5c25dd94422135b8302409adf252635a"
server_key_iv: "81c5cdfca3c911d1e42da2962376da7b"
server_key_version: "encryptionkey@0"
I20260923 21:03:59.527937 15580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:03:59.561226 15580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:03:59.566308 15580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:03:59.566672 15580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:03:59.567839 15580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:03:59.569383 15580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:03:59.569514 15580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:59.569679 15580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:03:59.569784 15580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:03:59.608031 15580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46123
I20260923 21:03:59.608093 15707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46123 every 8 connection(s)
I20260923 21:03:59.609750 15580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:03:59.610833 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 15580
I20260923 21:03:59.611675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43187
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44601
--webserver_port=34123
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:03:59.615751 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:59.613307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:54349)
0923 21:03:59.614198 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:59.614503 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":337,"threads_started":1}
W20260923 21:03:59.616575 15708 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44601 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44601: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
I20260923 21:03:59.617779 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:59.617288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:40885)
0923 21:03:59.617493 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:59.617592 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 21:03:59.618928 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:03:59.618525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:49231)
0923 21:03:59.618703 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 21:03:59.618818 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 21:03:59.619146 15708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 21:03:59.952168 15711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:03:59.952486 15711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:03:59.952584 15711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:03:59.960081 15711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:03:59.960213 15711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:03:59.960335 15711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:03:59.974598 15711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34123
--tserver_master_addrs=127.0.97.62:44601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:03:59.976052 15711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:03:59.977632 15711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:03:59.992991 15716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:59.993021 15717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:03:59.992941 15719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:03:59.993937 15711 server_base.cc:1034] running on GCE node
I20260923 21:03:59.994779 15711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:03:59.995879 15711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:03:59.997136 15711 hybrid_clock.cc:648] HybridClock initialized: now 1790197439997088 us; error 41 us; skew 500 ppm
I20260923 21:03:59.997490 15711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:03:59 dist-test-slave-8xn5 krb5kdc[14263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:00.000732 15711 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:04:00.002393 15711 webserver.cc:460] Webserver started at http://127.0.97.3:34123/ using document root <none> and password file <none>
I20260923 21:04:00.003146 15711 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:00.003301 15711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:00.009388 15711 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 21:04:00.012722 15726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:00.014317 15711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 21:04:00.014599 15711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "9aceb9d6306048389443d0e34a05848b"
format_stamp: "Formatted at 2026-09-23 21:03:53 on dist-test-slave-8xn5"
server_key: "0e379526ebaf6da5d07c236123b016ef"
server_key_iv: "bbc145666c708ab631e8c542c4b98973"
server_key_version: "encryptionkey@0"
I20260923 21:04:00.015676 15711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:00.063126 15711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:00.067840 15711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:00.068145 15711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:00.069197 15711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:04:00.070690 15711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:04:00.070809 15711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:00.070950 15711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:04:00.071038 15711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:00.110057 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:00.109167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:48001)
0923 21:04:00.109710 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:00.109814 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260923 21:04:00.111645 15711 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43187
I20260923 21:04:00.111675 15839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43187 every 8 connection(s)
I20260923 21:04:00.113253 15711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:04:00.118866 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:00.116605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:46985)
0923 21:04:00.117472 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:00.117712 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":334,"threads_started":1}
W20260923 21:04:00.119606 15840 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44601 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44601: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
I20260923 21:04:00.120532 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:00.120168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:57825)
0923 21:04:00.120319 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:00.120413 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 21:04:00.121551 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:00.121119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:55601)
0923 21:04:00.121314 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:00.121408 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 21:04:00.121950 15840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 21:04:00.123076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 15711
I20260923 21:04:00.620834 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:00.619864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:56267)
0923 21:04:00.620390 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:00.620511 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":212,"threads_started":1}
I20260923 21:04:01.112025 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:01.111230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:54989)
0923 21:04:01.111750 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:01.111857 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":258,"threads_started":1}
I20260923 21:04:01.123473 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:01.122609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:52031)
0923 21:04:01.123109 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:01.123219 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":224,"threads_started":1}
I20260923 21:04:01.622668 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:01.621789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:38425)
0923 21:04:01.622391 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:01.622476 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":243,"threads_started":1}
I20260923 21:04:02.114066 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:02.113073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:32983)
0923 21:04:02.113785 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:02.113887 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":315,"threads_started":1}
I20260923 21:04:02.125401 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:02.124586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:58109)
0923 21:04:02.125131 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:02.125215 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
I20260923 21:04:02.624354 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:02.623573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:56429)
0923 21:04:02.624069 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:02.624164 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":261,"threads_started":1}
I20260923 21:04:03.115899 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:03.115069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:44821)
0923 21:04:03.115599 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:03.115701 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":267,"threads_started":1}
I20260923 21:04:03.127236 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:03.126322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:59845)
0923 21:04:03.126915 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:03.127036 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":307,"threads_started":1}
I20260923 21:04:03.626312 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:03.625530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:36285)
0923 21:04:03.626042 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:03.626135 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":268,"threads_started":1}
I20260923 21:04:04.117882 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:04.117025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:45829)
0923 21:04:04.117622 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:04.117713 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":264,"threads_started":1}
I20260923 21:04:04.129021 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:04.128217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:33243)
0923 21:04:04.128749 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:04.128839 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":254,"threads_started":1}
I20260923 21:04:04.628439 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:04.627496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:33965)
0923 21:04:04.628087 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:04.628204 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":280,"threads_started":1}
I20260923 21:04:05.119953 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:05.119159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:49787)
0923 21:04:05.119688 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:05.119788 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":264,"threads_started":1}
I20260923 21:04:05.130990 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:05.130187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:52265)
0923 21:04:05.130725 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:05.130807 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":315,"threads_started":1}
I20260923 21:04:05.630221 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:05.629435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:36391)
0923 21:04:05.629942 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:05.630043 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":271,"threads_started":1}
I20260923 21:04:06.122020 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:06.121050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:59205)
0923 21:04:06.121713 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:06.121806 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":254,"threads_started":1}
I20260923 21:04:06.132941 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:06.131992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:59787)
0923 21:04:06.132664 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:06.132768 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":369,"threads_started":1}
I20260923 21:04:06.632228 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:06.631356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:32889)
0923 21:04:06.631913 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:06.632009 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":271,"threads_started":1}
I20260923 21:04:07.123947 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:07.123177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:39617)
0923 21:04:07.123681 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:07.123777 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":250,"threads_started":1}
I20260923 21:04:07.134985 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:07.134030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:39509)
0923 21:04:07.134665 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:07.134757 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":332,"threads_started":1}
I20260923 21:04:07.634006 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:07.633190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:49701)
0923 21:04:07.633689 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:07.633780 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260923 21:04:08.125914 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:08.125095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:60369)
0923 21:04:08.125641 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:08.125733 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":251,"threads_started":1}
I20260923 21:04:08.137286 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:08.136254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:42547)
0923 21:04:08.136930 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:08.137109 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260923 21:04:08.635773 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:08.634991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:34231)
0923 21:04:08.635482 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:08.635576 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":243,"threads_started":1}
I20260923 21:04:09.127837 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:09.127056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:42009)
0923 21:04:09.127574 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:09.127665 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260923 21:04:09.139078 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:09.138237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:53655)
0923 21:04:09.138788 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:09.138875 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":324,"threads_started":1}
I20260923 21:04:09.638052 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:09.637125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:35339)
0923 21:04:09.637744 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:09.637865 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":419,"threads_started":1}
I20260923 21:04:10.129731 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:10.128917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:42557)
0923 21:04:10.129455 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:10.129552 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":302,"threads_started":1}
I20260923 21:04:10.140969 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:10.140099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:57145)
0923 21:04:10.140698 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:10.140793 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":343,"threads_started":1}
I20260923 21:04:10.640143 15874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:10.639178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:56465)
0923 21:04:10.639708 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:10.639814 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":247,"threads_started":1}
I20260923 21:04:11.131700 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:11.130861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:52979)
0923 21:04:11.131337 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:11.131494 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":218,"threads_started":1}
I20260923 21:04:11.142776 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:11.141965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:49747)
0923 21:04:11.142498 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:11.142591 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":271,"threads_started":1}
I20260923 21:04:11.641947 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:11.641246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:45731)
0923 21:04:11.641696 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:11.641781 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260923 21:04:12.133869 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:12.132968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:54875)
0923 21:04:12.133538 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:12.133636 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":265,"threads_started":1}
I20260923 21:04:12.144701 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:12.143753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:40811)
0923 21:04:12.144428 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:12.144526 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":311,"threads_started":1}
I20260923 21:04:12.643909 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:12.643086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:45313)
0923 21:04:12.643609 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:12.643702 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":254,"threads_started":1}
I20260923 21:04:13.135819 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:13.134986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:46205)
0923 21:04:13.135492 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:13.135607 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
I20260923 21:04:13.146682 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:13.145776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:37401)
0923 21:04:13.146319 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:13.146478 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":247,"threads_started":1}
I20260923 21:04:13.645855 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:13.645014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:55569)
0923 21:04:13.645517 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:13.645607 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":281,"threads_started":1}
I20260923 21:04:14.137908 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:14.137058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:38305)
0923 21:04:14.137573 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:14.137673 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":269,"threads_started":1}
I20260923 21:04:14.148638 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:14.147775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:57631)
0923 21:04:14.148306 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:14.148430 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
I20260923 21:04:14.647912 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:14.647157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.2:55999)
0923 21:04:14.647620 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:14.647713 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260923 21:04:15.133073 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 15414
I20260923 21:04:15.140126 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:15.138955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.1:47591)
0923 21:04:15.139623 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:15.139771 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":270,"threads_started":1}
I20260923 21:04:15.150852 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:15.150079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44601 (local address 127.0.97.3:36077)
0923 21:04:15.150581 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:15.150661 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":286,"threads_started":1}
I20260923 21:04:15.154755 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:15.134524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.0.97.1:36223)
0923 21:04:15.135324 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:15.135337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:15.135376 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:04:15.136942 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:15.136951 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:15.137392 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:15.137693 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:15.139987 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:15.141618 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:15.142983 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:15.143720 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:15.143857 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:15.146984 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:15.146992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:15.147008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:15.147044 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:15.151375 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:15.152093 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:15.152100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:15.152108 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:15.152312 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:15.152701 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:15.152705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:15.152707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:15.153344 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:15.153580 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:15.154022 (+ 442us) server_negotiation.cc:299] Negotiation successful
0923 21:04:15.154456 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":479,"threads_started":1}
I20260923 21:04:15.156129 15572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:15.242296 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 15414
I20260923 21:04:15.260046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 15580
I20260923 21:04:15.280229 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:15.261157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.0.97.2:46123)
0923 21:04:15.261723 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:15.261743 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:15.261791 (+ 48us) server_negotiation.cc:407] Connection header received
0923 21:04:15.263521 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:15.263531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:15.263990 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:15.264398 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:15.265902 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:15.267299 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:15.268778 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:15.269509 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:15.269732 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:15.273123 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:15.273132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:15.273150 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:15.273214 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:15.276403 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:15.277239 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:15.277248 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:15.277256 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:15.277441 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:15.277917 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:15.277923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:15.277927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:15.278646 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:15.278967 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:15.279474 (+ 507us) server_negotiation.cc:299] Negotiation successful
0923 21:04:15.279925 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":258,"threads_started":1}
I20260923 21:04:15.281963 15703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:15.368798 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 15580
I20260923 21:04:15.387635 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 15711
I20260923 21:04:15.407815 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:15.388993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.0.97.3:43187)
0923 21:04:15.389615 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:15.389629 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:15.389671 (+ 42us) server_negotiation.cc:407] Connection header received
0923 21:04:15.391402 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:15.391411 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:15.391853 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:15.392154 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:15.393595 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:15.395097 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:15.396498 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:15.397261 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:15.397399 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:15.400404 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:15.400420 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:15.400441 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:15.400485 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:15.404050 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:15.405050 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:15.405059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:15.405069 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:15.405254 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:15.405648 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:15.405654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:15.405657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:15.406240 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:15.406563 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:15.407046 (+ 483us) server_negotiation.cc:299] Negotiation successful
0923 21:04:15.407510 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":314,"threads_started":1}
I20260923 21:04:15.409400 15834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:15.496407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 15711
2026-09-23T21:04:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25190 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:04:15 dist-test-slave-8xn5 krb5kdc[15897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:04:15 dist-test-slave-8xn5 krb5kdc[15897](info): set up 2 sockets
Sep 23 21:04:15 dist-test-slave-8xn5 krb5kdc[15897](info): commencing operation
krb5kdc: starting...
W20260923 21:04:16.679013 388 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:04:16 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:04:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:04:16.932785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45325
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:04:17.278745 15910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:17.279078 15910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:17.279179 15910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:17.286775 15910 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 21:04:17.286901 15910 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 21:04:17.286942 15910 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 21:04:17.286988 15910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:04:17.287035 15910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:17.287104 15910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:17.287158 15910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:04:17.287212 15910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:04:17.297742 15910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:17.299177 15910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:17.300524 15910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:17.313661 15918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:17.313587 15915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:17.313561 15916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:17.314468 15910 server_base.cc:1034] running on GCE node
I20260923 21:04:17.315374 15910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:17.317082 15910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:17.318411 15910 hybrid_clock.cc:648] HybridClock initialized: now 1790197457318262 us; error 64 us; skew 500 ppm
I20260923 21:04:17.318903 15910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:17 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:17.328778 15910 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:04:17.330708 15910 webserver.cc:460] Webserver started at http://127.0.97.62:44687/ using document root <none> and password file <none>
I20260923 21:04:17.331526 15910 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:17.331629 15910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:17.331941 15910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:04:17.335496 15910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "de2a29438d254ebbb1d0c880a341d7f4"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "6cee9f13ed384b30cb022e8b63f6516f"
server_key_iv: "05477a53338ce7a3672309ef1bafd714"
server_key_version: "encryptionkey@0"
I20260923 21:04:17.336553 15910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "de2a29438d254ebbb1d0c880a341d7f4"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "6cee9f13ed384b30cb022e8b63f6516f"
server_key_iv: "05477a53338ce7a3672309ef1bafd714"
server_key_version: "encryptionkey@0"
I20260923 21:04:17.342876 15910 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 21:04:17.347025 15925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:17.348661 15910 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:04:17.348873 15910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "de2a29438d254ebbb1d0c880a341d7f4"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "6cee9f13ed384b30cb022e8b63f6516f"
server_key_iv: "05477a53338ce7a3672309ef1bafd714"
server_key_version: "encryptionkey@0"
I20260923 21:04:17.349121 15910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:17.362547 15910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:17.367530 15910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:17.367902 15910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:17.387828 15910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45325
I20260923 21:04:17.387856 15976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45325 every 8 connection(s)
I20260923 21:04:17.389252 15910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:04:17.393257 15977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:04:17.397630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 15910
I20260923 21:04:17.397922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:04:17.398595 388 external_mini_cluster.cc:1428] Setting key 46c4b539c712611ae12804a149dc7b45
I20260923 21:04:17.405643 15977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Bootstrap starting.
I20260923 21:04:17.408962 15977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Neither blocks nor log segments found. Creating new log.
I20260923 21:04:17.410058 15977 log.cc:824] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:17.412962 15977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: No bootstrap required, opened a new log
Sep 23 21:04:17 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:04:17.422650 15977 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:17.423056 15977 raft_consensus.cc:374] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:04:17.423177 15977 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2a29438d254ebbb1d0c880a341d7f4, State: Initialized, Role: FOLLOWER
I20260923 21:04:17.423446 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:17.400699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.0.97.62:45325)
0923 21:04:17.401582 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:17.401595 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:17.401639 (+ 44us) server_negotiation.cc:407] Connection header received
0923 21:04:17.404134 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:17.404161 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:17.404583 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:17.404987 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:17.406721 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:17.408071 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:17.409543 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:17.410313 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:17.410554 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:17.415234 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:17.415242 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:17.415261 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:17.415341 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:17.418722 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:17.419663 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:17.419669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:17.419679 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:17.419874 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:17.420296 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:17.420299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:17.420301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:17.420886 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:17.421106 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:17.421606 (+ 500us) server_negotiation.cc:299] Negotiation successful
0923 21:04:17.422015 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":408,"threads_started":1}
I20260923 21:04:17.423899 15977 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:17.424178 15977 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:04:17.424327 15977 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:04:17.424515 15977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:04:17.426960 15977 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:17.427464 15977 leader_election.cc:302] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2a29438d254ebbb1d0c880a341d7f4; no voters:
I20260923 21:04:17.428118 15977 leader_election.cc:288] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:04:17.428280 15982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:04:17.428992 15982 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 LEADER]: Becoming Leader. State: Replica: de2a29438d254ebbb1d0c880a341d7f4, State: Running, Role: LEADER
I20260923 21:04:17.429538 15982 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:17.430299 15977 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:04:17.432821 15984 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2a29438d254ebbb1d0c880a341d7f4. Latest consensus state: current_term: 1 leader_uuid: "de2a29438d254ebbb1d0c880a341d7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } } }
I20260923 21:04:17.433337 15984 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:04:17.434211 15983 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de2a29438d254ebbb1d0c880a341d7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } } }
I20260923 21:04:17.434834 15983 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:04:17.435048 15989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:04:17.441254 15989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:04:17.451474 15989 catalog_manager.cc:1347] Generated new cluster ID: 68db847905e845e183c20c861ba72811
I20260923 21:04:17.451632 15989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:04:17.464982 15989 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:04:17.467060 15989 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:04:17.475549 15989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Generated new TSK 0
I20260923 21:04:17.476713 15989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:04:17.546882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45325
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:04:17.893044 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:17.893399 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:17.893575 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:17.900993 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:17.901180 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:17.901364 16003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:04:17.911847 16003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:17.913259 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:17.914637 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:17.928450 16008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:17.928427 16009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:17.928485 16011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:17.929373 16003 server_base.cc:1034] running on GCE node
I20260923 21:04:17.930068 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:17.931185 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:17.932423 16003 hybrid_clock.cc:648] HybridClock initialized: now 1790197457932372 us; error 73 us; skew 500 ppm
I20260923 21:04:17.932781 16003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:17 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:17.936244 16003 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:04:17.938022 16003 webserver.cc:460] Webserver started at http://127.0.97.1:34331/ using document root <none> and password file <none>
I20260923 21:04:17.938831 16003 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:17.938984 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:17.939316 16003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:04:17.942478 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "5625f3fe57844f31bd14aff52ddb1a4f"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "adaeea6fbccca01c5b43fa83a3d386f2"
server_key_iv: "e1cec6a2f9eb8fd76cf33f433b2995ce"
server_key_version: "encryptionkey@0"
I20260923 21:04:17.943302 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5625f3fe57844f31bd14aff52ddb1a4f"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "adaeea6fbccca01c5b43fa83a3d386f2"
server_key_iv: "e1cec6a2f9eb8fd76cf33f433b2995ce"
server_key_version: "encryptionkey@0"
I20260923 21:04:17.948889 16003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:04:17.952773 16018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:17.954581 16003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:04:17.954790 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "5625f3fe57844f31bd14aff52ddb1a4f"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "adaeea6fbccca01c5b43fa83a3d386f2"
server_key_iv: "e1cec6a2f9eb8fd76cf33f433b2995ce"
server_key_version: "encryptionkey@0"
I20260923 21:04:17.955017 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:17.985512 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:17.990266 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:17.990617 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:17.991732 16003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:04:17.993171 16003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:04:17.993279 16003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:17.993432 16003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:04:17.993521 16003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:18.028383 16003 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43539
I20260923 21:04:18.028431 16130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43539 every 8 connection(s)
I20260923 21:04:18.029783 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:04:18.034857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16003
I20260923 21:04:18.035084 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:04:18.035912 388 external_mini_cluster.cc:1428] Setting key 8784c04596e68a367169d0a989f9acd8
Sep 23 21:04:18 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:04:18.053541 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:18.032677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37631 (local address 127.0.97.62:45325)
0923 21:04:18.033179 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:18.033186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:18.034248 (+ 1062us) server_negotiation.cc:407] Connection header received
0923 21:04:18.037020 (+ 2772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:18.037024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:18.037165 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:18.037297 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:18.039306 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.040056 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.041714 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.042131 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.042286 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:18.046417 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:18.046422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:18.046426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:18.046464 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:18.049383 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:18.050216 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:18.050221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:18.050223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:18.050410 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:18.050865 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:18.050870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:18.050874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:18.051165 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:18.051331 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:18.052355 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0923 21:04:18.052795 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":235,"threads_started":1}
I20260923 21:04:18.054208 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:18.033329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:37631)
0923 21:04:18.034069 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:18.034112 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:18.036708 (+ 2596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:18.037561 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:18.037570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:18.038462 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:04:18.039101 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:18.039120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.040225 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.040231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:18.041538 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:18.041550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.042414 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.042423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:18.042691 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:18.043427 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:04:18.043450 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:04:18.046095 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:04:18.049565 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:18.049575 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:18.049600 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:18.050050 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:18.050534 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:18.050545 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:18.050548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:18.050742 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:18.051303 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:04:18.051312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:04:18.051770 (+ 458us) client_negotiation.cc:769] Sending connection context
0923 21:04:18.052114 (+ 344us) client_negotiation.cc:240] Negotiation successful
0923 21:04:18.052694 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":279,"threads_started":1}
I20260923 21:04:18.056114 16131 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45325
I20260923 21:04:18.056475 16131 heartbeater.cc:464] Registering TS with master...
I20260923 21:04:18.057498 16131 heartbeater.cc:511] Master 127.0.97.62:45325 requested a full tablet report, sending...
I20260923 21:04:18.060240 15942 ts_manager.cc:194] Registered new tserver with Master: 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:04:18.061899 15942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:04:18.094292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45325
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:04:18.441059 16137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:18.441581 16137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:18.441792 16137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:18.450579 16137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:18.450727 16137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:18.450920 16137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:04:18.462309 16137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:18.463752 16137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:18.465132 16137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:18.479343 16145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:18.479385 16143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:18.479300 16142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:18.480795 16137 server_base.cc:1034] running on GCE node
I20260923 21:04:18.481523 16137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:18.482816 16137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:18.484092 16137 hybrid_clock.cc:648] HybridClock initialized: now 1790197458484046 us; error 51 us; skew 500 ppm
I20260923 21:04:18.484449 16137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:18 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:18.487967 16137 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:04:18.489544 16137 webserver.cc:460] Webserver started at http://127.0.97.2:41669/ using document root <none> and password file <none>
I20260923 21:04:18.490212 16137 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:18.490322 16137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:18.490675 16137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:04:18.493898 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "800274ba5ea14fd29c64f3cf09fbcbf0"
format_stamp: "Formatted at 2026-09-23 21:04:18 on dist-test-slave-8xn5"
server_key: "bf46ecbe8c126413270537ecc30164dd"
server_key_iv: "937f2f86cac08df835cba2d5dbb0acb7"
server_key_version: "encryptionkey@0"
I20260923 21:04:18.494729 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "800274ba5ea14fd29c64f3cf09fbcbf0"
format_stamp: "Formatted at 2026-09-23 21:04:18 on dist-test-slave-8xn5"
server_key: "bf46ecbe8c126413270537ecc30164dd"
server_key_iv: "937f2f86cac08df835cba2d5dbb0acb7"
server_key_version: "encryptionkey@0"
I20260923 21:04:18.500279 16137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:04:18.503826 16152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:18.505264 16137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 21:04:18.505443 16137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "800274ba5ea14fd29c64f3cf09fbcbf0"
format_stamp: "Formatted at 2026-09-23 21:04:18 on dist-test-slave-8xn5"
server_key: "bf46ecbe8c126413270537ecc30164dd"
server_key_iv: "937f2f86cac08df835cba2d5dbb0acb7"
server_key_version: "encryptionkey@0"
I20260923 21:04:18.505669 16137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:18.519109 16137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:18.523871 16137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:18.524194 16137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:18.525332 16137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:04:18.526819 16137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:04:18.526894 16137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:18.527037 16137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:04:18.527100 16137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:18.565025 16137 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45893
I20260923 21:04:18.565074 16264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45893 every 8 connection(s)
I20260923 21:04:18.566740 16137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:04:18.569461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16137
I20260923 21:04:18.569665 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:04:18.570289 388 external_mini_cluster.cc:1428] Setting key 956cc694a6384e390d2f1dc6e92b4ef7
Sep 23 21:04:18 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:04:18.589083 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:18.569759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45975 (local address 127.0.97.62:45325)
0923 21:04:18.570269 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:18.570275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:18.571360 (+ 1085us) server_negotiation.cc:407] Connection header received
0923 21:04:18.573259 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:18.573269 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:18.573361 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:18.573483 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:18.575179 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.576372 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.578070 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.578882 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.579025 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:18.583214 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:18.583219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:18.583223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:18.583262 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:18.585837 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:18.586701 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:18.586706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:18.586708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:18.586844 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:18.587298 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:18.587307 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:18.587310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:18.587593 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:18.587761 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:18.588459 (+ 698us) server_negotiation.cc:299] Negotiation successful
0923 21:04:18.588693 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
I20260923 21:04:18.590082 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:18.570283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:45975)
0923 21:04:18.571195 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:18.571251 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:18.572981 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:18.573788 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:18.573798 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:18.574392 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:04:18.574998 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:18.575018 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.576542 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.576545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:18.577914 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:18.577931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:18.579097 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:18.579102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:18.579304 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:18.580050 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:04:18.580081 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:04:18.582974 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:04:18.585996 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:18.586005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:18.586024 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:18.586566 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:18.586956 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:18.586962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:18.586965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:18.587155 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:18.587722 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:04:18.587730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:04:18.588080 (+ 350us) client_negotiation.cc:769] Sending connection context
0923 21:04:18.588367 (+ 287us) client_negotiation.cc:240] Negotiation successful
0923 21:04:18.588915 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":370,"threads_started":1}
I20260923 21:04:18.591711 16265 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45325
I20260923 21:04:18.592073 16265 heartbeater.cc:464] Registering TS with master...
I20260923 21:04:18.593012 16265 heartbeater.cc:511] Master 127.0.97.62:45325 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:04:18.594776 15942 ts_manager.cc:194] Registered new tserver with Master: 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
I20260923 21:04:18.595629 15942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:04:18.630237 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45325
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:04:18.972240 16271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:18.972533 16271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:18.972718 16271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:18.980536 16271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:18.980664 16271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:18.980842 16271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:04:18.990984 16271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:18.992532 16271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:18.993911 16271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:19.007467 16276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:19.007433 16277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:19.008036 16279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:19.008464 16271 server_base.cc:1034] running on GCE node
I20260923 21:04:19.009081 16271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:19.010288 16271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:19.011574 16271 hybrid_clock.cc:648] HybridClock initialized: now 1790197459011461 us; error 112 us; skew 500 ppm
I20260923 21:04:19.011982 16271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:19 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:19.015559 16271 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:04:19.017313 16271 webserver.cc:460] Webserver started at http://127.0.97.3:39087/ using document root <none> and password file <none>
I20260923 21:04:19.018062 16271 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:19.018200 16271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:19.018595 16271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:04:19.021663 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "5a13b2881b2048998fbbc0d63be1ac23"
format_stamp: "Formatted at 2026-09-23 21:04:19 on dist-test-slave-8xn5"
server_key: "a3681a113e6798b584b8a8e7e16c28b1"
server_key_iv: "a91d798f823b438be5d568982ea425b1"
server_key_version: "encryptionkey@0"
I20260923 21:04:19.022535 16271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5a13b2881b2048998fbbc0d63be1ac23"
format_stamp: "Formatted at 2026-09-23 21:04:19 on dist-test-slave-8xn5"
server_key: "a3681a113e6798b584b8a8e7e16c28b1"
server_key_iv: "a91d798f823b438be5d568982ea425b1"
server_key_version: "encryptionkey@0"
I20260923 21:04:19.027974 16271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:04:19.031847 16286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:19.033370 16271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:04:19.033537 16271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "5a13b2881b2048998fbbc0d63be1ac23"
format_stamp: "Formatted at 2026-09-23 21:04:19 on dist-test-slave-8xn5"
server_key: "a3681a113e6798b584b8a8e7e16c28b1"
server_key_iv: "a91d798f823b438be5d568982ea425b1"
server_key_version: "encryptionkey@0"
I20260923 21:04:19.033732 16271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:19.048034 16271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:19.052852 16271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:19.053171 16271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:19.054396 16271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:04:19.056218 16271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:04:19.056313 16271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:19.056448 16271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:04:19.056514 16271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:19.065578 16131 heartbeater.cc:503] Master 127.0.97.62:45325 was elected leader, sending a full tablet report...
I20260923 21:04:19.095029 16271 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37535
I20260923 21:04:19.095142 16398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37535 every 8 connection(s)
I20260923 21:04:19.096388 16271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:04:19.104763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16271
I20260923 21:04:19.105017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:04:19.105610 388 external_mini_cluster.cc:1428] Setting key 8942303b144db29fae9282cdcb46029b
Sep 23 21:04:19 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:04:19.119554 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.099259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54765 (local address 127.0.97.62:45325)
0923 21:04:19.099878 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.099884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.100619 (+ 735us) server_negotiation.cc:407] Connection header received
0923 21:04:19.102907 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.102915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.103028 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.103180 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:19.105318 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.106189 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.108722 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.109321 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.109478 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.113323 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:19.113328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:19.113332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:19.113367 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:19.116280 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.117061 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.117065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.117068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.117233 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.117679 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.117685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.117689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.118005 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:19.118139 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.119048 (+ 909us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.119264 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":388,"threads_started":1}
I20260923 21:04:19.120767 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.099623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:54765)
0923 21:04:19.100434 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.100475 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.102568 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.103464 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.103474 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.104108 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:04:19.105089 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.105122 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.106508 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.106515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.108548 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.108566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.109563 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.109570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.109928 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.110816 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:04:19.110840 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:04:19.113098 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:04:19.116442 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:19.116449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:19.116464 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:19.116911 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:19.117335 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:19.117339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:19.117341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:19.117527 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:19.118142 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:04:19.118148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:04:19.118667 (+ 519us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.118900 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.119353 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":319,"threads_started":1}
I20260923 21:04:19.122387 16399 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45325
I20260923 21:04:19.122828 16399 heartbeater.cc:464] Registering TS with master...
I20260923 21:04:19.123678 16399 heartbeater.cc:511] Master 127.0.97.62:45325 requested a full tablet report, sending...
I20260923 21:04:19.125254 15941 ts_manager.cc:194] Registered new tserver with Master: 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535)
I20260923 21:04:19.126180 15941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54765
I20260923 21:04:19.135078 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:04:19.154016 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.140175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.97.62:45325)
0923 21:04:19.140382 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.140386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.140903 (+ 517us) server_negotiation.cc:407] Connection header received
0923 21:04:19.141059 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.141063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.141132 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.141226 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:19.142925 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.143774 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.145285 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.145683 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.145830 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.148441 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:19.148446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:19.148449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:19.148481 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:19.151231 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.151974 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.151978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.151981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.152097 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.152452 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.152455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.152457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.152911 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:19.153099 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.153447 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.153621 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 21:04:19.158900 15941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:04:19.165894 15941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:04:19.204453 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.179462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:38880)
0923 21:04:19.193891 (+ 14429us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.193928 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.194160 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.196395 (+ 2235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.196407 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.196452 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:19.196985 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.197002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.199020 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.199025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.201037 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.201055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.203461 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.203468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.203661 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.203715 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.203870 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.204068 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14194,"thread_start_us":17040,"threads_started":1}
I20260923 21:04:19.204452 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.180251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.0.97.1:43539)
0923 21:04:19.195346 (+ 15095us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.195355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.195392 (+ 37us) server_negotiation.cc:407] Connection header received
0923 21:04:19.195506 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.195514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.195874 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.196111 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:19.197195 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.198856 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.201294 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.202919 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.203059 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.203345 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.203941 (+ 596us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.204133 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14887,"thread_start_us":14744,"threads_started":1}
I20260923 21:04:19.205833 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.179141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37535 (local address 127.0.0.1:57258)
0923 21:04:19.180716 (+ 1575us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.180769 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.180948 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.197928 (+ 16980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.197935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.197960 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:19.198603 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.198613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.200849 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.200857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.203062 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.203075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.204928 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.204935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.205096 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.205128 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.205206 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.205332 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"mutex_wait_us":29,"thread_start_us":1156,"threads_started":1}
I20260923 21:04:19.206689 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.179452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.0.97.3:37535)
0923 21:04:19.197395 (+ 17943us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.197402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.197433 (+ 31us) server_negotiation.cc:407] Connection header received
0923 21:04:19.197514 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.197519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.197681 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.198884 (+ 1203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:19.199173 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.200703 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.203281 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.204684 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.205813 (+ 1129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.206047 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.206132 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.206297 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17776,"thread_start_us":17612,"threads_started":1}
I20260923 21:04:19.207141 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.181173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45893 (local address 127.0.0.1:60062)
0923 21:04:19.196853 (+ 15680us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.196874 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.197021 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.197865 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.197872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.197905 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:19.198598 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.198613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.200684 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.200690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.203162 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.203178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.205431 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.205437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.205622 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.205662 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.206437 (+ 775us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.206576 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15500,"mutex_wait_us":110,"thread_start_us":13652,"threads_started":1}
I20260923 21:04:19.207931 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.181555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.0.97.2:45893)
0923 21:04:19.194789 (+ 13234us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.194798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.197175 (+ 2377us) server_negotiation.cc:407] Connection header received
0923 21:04:19.197296 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.197303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.197610 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.197843 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:19.198883 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.200518 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.203352 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.205220 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.205379 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.205582 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.206518 (+ 936us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.207577 (+ 1059us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13075,"thread_start_us":12924,"threads_started":1}
I20260923 21:04:19.209093 16066 tablet_service.cc:1467] Processing CreateTablet for tablet 14426bd86b294dfdb12251e612f09d7c (DEFAULT_TABLE table=test-table [id=c4c1e6ed38b946c7ad15f00f2973fc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:04:19.210110 16334 tablet_service.cc:1467] Processing CreateTablet for tablet 14426bd86b294dfdb12251e612f09d7c (DEFAULT_TABLE table=test-table [id=c4c1e6ed38b946c7ad15f00f2973fc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:04:19.210783 16066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14426bd86b294dfdb12251e612f09d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:04:19.211364 16334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14426bd86b294dfdb12251e612f09d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:04:19.211421 16200 tablet_service.cc:1467] Processing CreateTablet for tablet 14426bd86b294dfdb12251e612f09d7c (DEFAULT_TABLE table=test-table [id=c4c1e6ed38b946c7ad15f00f2973fc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:04:19.212687 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14426bd86b294dfdb12251e612f09d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:04:19.223676 16418 tablet_bootstrap.cc:492] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Bootstrap starting.
I20260923 21:04:19.227111 16418 tablet_bootstrap.cc:654] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Neither blocks nor log segments found. Creating new log.
I20260923 21:04:19.228381 16418 log.cc:824] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:19.231328 16419 tablet_bootstrap.cc:492] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Bootstrap starting.
I20260923 21:04:19.233385 16420 tablet_bootstrap.cc:492] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Bootstrap starting.
I20260923 21:04:19.235198 16418 tablet_bootstrap.cc:492] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: No bootstrap required, opened a new log
I20260923 21:04:19.235529 16418 ts_tablet_manager.cc:1397] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260923 21:04:19.235678 16419 tablet_bootstrap.cc:654] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Neither blocks nor log segments found. Creating new log.
I20260923 21:04:19.237118 16420 tablet_bootstrap.cc:654] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Neither blocks nor log segments found. Creating new log.
I20260923 21:04:19.237321 16419 log.cc:824] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:19.238458 16420 log.cc:824] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:19.241350 16420 tablet_bootstrap.cc:492] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: No bootstrap required, opened a new log
I20260923 21:04:19.241559 16420 ts_tablet_manager.cc:1397] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 21:04:19.244189 16419 tablet_bootstrap.cc:492] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: No bootstrap required, opened a new log
I20260923 21:04:19.244469 16419 ts_tablet_manager.cc:1397] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20260923 21:04:19.244195 16418 raft_consensus.cc:348] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.244637 16418 raft_consensus.cc:374] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:04:19.244762 16418 raft_consensus.cc:729] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a13b2881b2048998fbbc0d63be1ac23, State: Initialized, Role: FOLLOWER
I20260923 21:04:19.245389 16418 consensus_queue.cc:260] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.246560 16399 heartbeater.cc:503] Master 127.0.97.62:45325 was elected leader, sending a full tablet report...
I20260923 21:04:19.247200 16418 ts_tablet_manager.cc:1428] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 21:04:19.252386 16420 raft_consensus.cc:348] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.252856 16420 raft_consensus.cc:374] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:04:19.252951 16420 raft_consensus.cc:729] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800274ba5ea14fd29c64f3cf09fbcbf0, State: Initialized, Role: FOLLOWER
I20260923 21:04:19.253628 16420 consensus_queue.cc:260] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.254791 16265 heartbeater.cc:503] Master 127.0.97.62:45325 was elected leader, sending a full tablet report...
I20260923 21:04:19.255052 16419 raft_consensus.cc:348] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.255527 16419 raft_consensus.cc:374] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:04:19.255551 16420 ts_tablet_manager.cc:1428] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260923 21:04:19.255641 16419 raft_consensus.cc:729] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5625f3fe57844f31bd14aff52ddb1a4f, State: Initialized, Role: FOLLOWER
I20260923 21:04:19.256166 16419 consensus_queue.cc:260] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.257898 16419 ts_tablet_manager.cc:1428] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
W20260923 21:04:19.282698 16132 tablet.cc:2369] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:04:19.319041 16266 tablet.cc:2369] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:04:19.348104 16400 tablet.cc:2369] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:04:19.397436 16425 raft_consensus.cc:482] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:04:19.397782 16425 raft_consensus.cc:504] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.399977 16425 leader_election.cc:288] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539)
I20260923 21:04:19.407102 16426 raft_consensus.cc:482] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:04:19.407718 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.400604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41875 (local address 127.0.97.1:43539)
0923 21:04:19.400818 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.400824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.401518 (+ 694us) server_negotiation.cc:407] Connection header received
0923 21:04:19.401876 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.401883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.401987 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.402093 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:19.402734 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.403804 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.405634 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.406572 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.406742 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.406857 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.407223 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.407455 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 21:04:19.408061 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.400375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37535 (local address 127.0.97.2:50495)
0923 21:04:19.401853 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.401872 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.402452 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.402876 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.402882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.402911 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:19.403188 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.403196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.404396 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.404406 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.405856 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.405869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.407195 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.407201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.407406 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.407451 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.407548 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.407684 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"thread_start_us":626,"threads_started":1}
I20260923 21:04:19.408085 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.401082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50495 (local address 127.0.97.3:37535)
0923 21:04:19.401306 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.401313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.401983 (+ 670us) server_negotiation.cc:407] Connection header received
0923 21:04:19.402626 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.402654 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.402737 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.402837 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:19.403346 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.404254 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.406009 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.406982 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.407123 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.407244 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.407584 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.407711 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 21:04:19.407467 16426 raft_consensus.cc:504] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.408396 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.400290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.97.2:41875)
0923 21:04:19.401387 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.401407 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.401683 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.402119 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.402124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.402153 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:19.402551 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.402563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.403945 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.403949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.405429 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.405442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.406813 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.406820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.407005 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.407042 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.407522 (+ 480us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.407677 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":1201,"threads_started":1}
I20260923 21:04:19.409287 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14426bd86b294dfdb12251e612f09d7c" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a13b2881b2048998fbbc0d63be1ac23" is_pre_election: true
I20260923 21:04:19.409541 16086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14426bd86b294dfdb12251e612f09d7c" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" is_pre_election: true
I20260923 21:04:19.410060 16354 raft_consensus.cc:2454] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 0.
I20260923 21:04:19.410199 16086 raft_consensus.cc:2454] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 0.
I20260923 21:04:19.410358 16426 leader_election.cc:288] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
I20260923 21:04:19.411008 16153 leader_election.cc:302] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a13b2881b2048998fbbc0d63be1ac23, 800274ba5ea14fd29c64f3cf09fbcbf0; no voters:
I20260923 21:04:19.411451 16425 raft_consensus.cc:2781] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:04:19.411605 16425 raft_consensus.cc:482] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:04:19.411716 16425 raft_consensus.cc:3037] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:04:19.415350 16425 raft_consensus.cc:504] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.416617 16354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14426bd86b294dfdb12251e612f09d7c" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a13b2881b2048998fbbc0d63be1ac23"
I20260923 21:04:19.416625 16086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14426bd86b294dfdb12251e612f09d7c" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5625f3fe57844f31bd14aff52ddb1a4f"
I20260923 21:04:19.416867 16086 raft_consensus.cc:3037] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:04:19.416873 16354 raft_consensus.cc:3037] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:04:19.419490 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.410525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37535 (local address 127.0.97.1:50677)
0923 21:04:19.411350 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.411372 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.411548 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.412196 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.412204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.412255 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:19.412714 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.412723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.414018 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.414023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.416361 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.416374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.418455 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.418461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.418675 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.418716 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.418837 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.419003 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":8,"thread_start_us":506,"threads_started":1}
I20260923 21:04:19.419572 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.411878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45933 (local address 127.0.97.2:45893)
0923 21:04:19.412073 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.412080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.412943 (+ 863us) server_negotiation.cc:407] Connection header received
0923 21:04:19.413104 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.413124 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.413212 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.413318 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:19.413915 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.414772 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.416969 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.418235 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.418386 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.418479 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.419154 (+ 675us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.419301 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:04:19.419806 16425 leader_election.cc:288] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 election: Requested vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539)
I20260923 21:04:19.420239 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.410789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50677 (local address 127.0.97.3:37535)
0923 21:04:19.410964 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.410970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.411444 (+ 474us) server_negotiation.cc:407] Connection header received
0923 21:04:19.411730 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.411735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.411836 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.411920 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:19.412860 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.413873 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.416548 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.417978 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.418097 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.418226 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.419807 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.419958 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 21:04:19.420331 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.411493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45893 (local address 127.0.97.1:45933)
0923 21:04:19.412818 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:19.412838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:19.412977 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:19.413350 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:19.413354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:19.413382 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:19.413737 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.413748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.414945 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.414950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.416771 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:19.416787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.418720 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.418727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:19.418935 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.418975 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 21:04:19.419769 (+ 794us) client_negotiation.cc:240] Negotiation successful
0923 21:04:19.419923 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":2,"thread_start_us":806,"threads_started":1}
I20260923 21:04:19.420585 16354 raft_consensus.cc:2454] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 1.
I20260923 21:04:19.420930 16353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14426bd86b294dfdb12251e612f09d7c" candidate_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a13b2881b2048998fbbc0d63be1ac23" is_pre_election: true
I20260923 21:04:19.421182 16086 raft_consensus.cc:2454] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 1.
I20260923 21:04:19.421392 16353 raft_consensus.cc:2379] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5625f3fe57844f31bd14aff52ddb1a4f in current term 1: Already voted for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in this term.
I20260923 21:04:19.421247 16220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14426bd86b294dfdb12251e612f09d7c" candidate_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" is_pre_election: true
I20260923 21:04:19.421706 16154 leader_election.cc:302] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f, 800274ba5ea14fd29c64f3cf09fbcbf0; no voters:
I20260923 21:04:19.421826 16220 raft_consensus.cc:2379] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5625f3fe57844f31bd14aff52ddb1a4f in current term 1: Already voted for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in this term.
I20260923 21:04:19.422097 16425 raft_consensus.cc:2781] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:04:19.422645 16425 raft_consensus.cc:686] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 LEADER]: Becoming Leader. State: Replica: 800274ba5ea14fd29c64f3cf09fbcbf0, State: Running, Role: LEADER
I20260923 21:04:19.422838 16019 leader_election.cc:302] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f; no voters: 5a13b2881b2048998fbbc0d63be1ac23, 800274ba5ea14fd29c64f3cf09fbcbf0
I20260923 21:04:19.423308 16426 raft_consensus.cc:2726] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 21:04:19.423249 16425 consensus_queue.cc:237] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:19.428817 15942 catalog_manager.cc:5495] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } health_report { overall_health: HEALTHY } } }
I20260923 21:04:19.450722 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.445199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.0.97.2:45893)
0923 21:04:19.445417 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.445423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.445443 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:04:19.445653 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.445659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.445767 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.445930 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:04:19.446315 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.447074 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.448473 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.448908 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.449043 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.449549 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:04:19.450080 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:04:19.450256 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.450306 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.450412 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 21:04:19.459405 16354 raft_consensus.cc:1261] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Refusing update from remote peer 800274ba5ea14fd29c64f3cf09fbcbf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:04:19.459438 16086 raft_consensus.cc:1261] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Refusing update from remote peer 800274ba5ea14fd29c64f3cf09fbcbf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:04:19.460438 16431 consensus_queue.cc:1035] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:04:19.460829 16425 consensus_queue.cc:1035] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:04:19.473937 16435 mvcc.cc:205] Tried to move back new op lower bound from 7332648793935400960 to 7332648793801957376. Current Snapshot: MvccSnapshot[applied={T|T < 7332648793935400960}]
I20260923 21:04:19.491142 15942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:04:19.491452 15942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:04:19.495210 15942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Sending DeleteTablet for 3 replicas of tablet 14426bd86b294dfdb12251e612f09d7c
I20260923 21:04:19.496557 16066 tablet_service.cc:1514] Processing DeleteTablet for tablet 14426bd86b294dfdb12251e612f09d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:04:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38880
I20260923 21:04:19.496779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 16003
I20260923 21:04:19.497001 16334 tablet_service.cc:1514] Processing DeleteTablet for tablet 14426bd86b294dfdb12251e612f09d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:04:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57258
I20260923 21:04:19.497742 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet 14426bd86b294dfdb12251e612f09d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:04:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60062
I20260923 21:04:19.510468 16444 tablet_replica.cc:330] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: stopping tablet replica
I20260923 21:04:19.510515 16447 tablet_replica.cc:330] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: stopping tablet replica
I20260923 21:04:19.510962 16447 raft_consensus.cc:2229] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:04:19.511013 16444 raft_consensus.cc:2229] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:04:19.511677 16447 raft_consensus.cc:2258] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:04:19.511966 16444 raft_consensus.cc:2258] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:04:19.512251 16445 tablet_replica.cc:330] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: stopping tablet replica
I20260923 21:04:19.512723 16445 raft_consensus.cc:2229] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:04:19.513247 16445 raft_consensus.cc:2258] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:04:19.514412 16447 ts_tablet_manager.cc:1905] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:04:19.515635 16444 ts_tablet_manager.cc:1905] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:04:19.517115 16445 ts_tablet_manager.cc:1905] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:04:19.523301 16444 ts_tablet_manager.cc:1918] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:04:19.523509 16444 log.cc:1196] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/14426bd86b294dfdb12251e612f09d7c
I20260923 21:04:19.524030 16444 ts_tablet_manager.cc:1939] T 14426bd86b294dfdb12251e612f09d7c P 5625f3fe57844f31bd14aff52ddb1a4f: Deleting consensus metadata
I20260923 21:04:19.525393 16445 ts_tablet_manager.cc:1918] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:04:19.525655 16445 log.cc:1196] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/14426bd86b294dfdb12251e612f09d7c
I20260923 21:04:19.526185 15927 catalog_manager.cc:4841] TS 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539): tablet 14426bd86b294dfdb12251e612f09d7c (table test-table [id=c4c1e6ed38b946c7ad15f00f2973fc49]) successfully deleted
I20260923 21:04:19.526840 16445 ts_tablet_manager.cc:1939] T 14426bd86b294dfdb12251e612f09d7c P 5a13b2881b2048998fbbc0d63be1ac23: Deleting consensus metadata
I20260923 21:04:19.528705 16447 ts_tablet_manager.cc:1918] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:04:19.528996 16447 log.cc:1196] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/14426bd86b294dfdb12251e612f09d7c
I20260923 21:04:19.529036 15926 catalog_manager.cc:4841] TS 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535): tablet 14426bd86b294dfdb12251e612f09d7c (table test-table [id=c4c1e6ed38b946c7ad15f00f2973fc49]) successfully deleted
I20260923 21:04:19.529721 16447 ts_tablet_manager.cc:1939] T 14426bd86b294dfdb12251e612f09d7c P 800274ba5ea14fd29c64f3cf09fbcbf0: Deleting consensus metadata
Sep 23 21:04:19 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:04:19.531980 15926 catalog_manager.cc:4841] TS 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893): tablet 14426bd86b294dfdb12251e612f09d7c (table test-table [id=c4c1e6ed38b946c7ad15f00f2973fc49]) successfully deleted
I20260923 21:04:19.540002 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.498147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.0.97.1:43539)
0923 21:04:19.498407 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.498417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.509578 (+ 11161us) server_negotiation.cc:407] Connection header received
0923 21:04:19.513136 (+ 3558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.513142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.513246 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.513333 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:19.515991 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.517226 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.519551 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.520227 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.520379 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.531809 (+ 11430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:19.531831 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:19.531853 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:19.531936 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:19.535897 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.537358 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.537369 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.537373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.537547 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.538172 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.538177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.538181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.538710 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:19.539024 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.539366 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.539616 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 21:04:19.540788 16126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:19.673859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 16003
I20260923 21:04:19.691668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 16137
Sep 23 21:04:19 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:04:19.711054 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.692525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.0.97.2:45893)
0923 21:04:19.692871 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.692877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.692898 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:04:19.692964 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.692969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.693048 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.693162 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:19.694606 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.695805 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.697325 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.698026 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.698158 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.703128 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:19.703143 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:19.703163 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:19.703221 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:19.707475 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.708434 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.708443 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.708447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.708632 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.709165 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.709171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.709174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.709693 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:19.710088 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.710437 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.710674 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260923 21:04:19.711858 16260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:19.830202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 16137
I20260923 21:04:19.850050 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 16271
Sep 23 21:04:19 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:04:19.865327 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:19.850931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57270 (local address 127.0.97.3:37535)
0923 21:04:19.851098 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:19.851102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:19.851120 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:04:19.851266 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:19.851270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:19.851334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:19.851421 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:19.852824 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.853610 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.855178 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:19.855585 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:19.855678 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:19.860182 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:19.860208 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:19.860224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:19.860265 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:19.862955 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.863613 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.863619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.863622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.863746 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:19.864062 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:19.864065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:19.864067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:19.864421 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:19.864635 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:19.864920 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 21:04:19.865084 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 21:04:19.865993 16394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:19.956526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 16271
I20260923 21:04:19.973994 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 15910
I20260923 21:04:19.975356 15972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:20.089563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 15910
I20260923 21:04:40.111637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45325
--webserver_interface=127.0.97.62
--webserver_port=44687
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:04:40.469242 16452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:40.469647 16452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:40.469827 16452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:40.477176 16452 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 21:04:40.477345 16452 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 21:04:40.477444 16452 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 21:04:40.477541 16452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:04:40.477622 16452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:40.477727 16452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:40.477819 16452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:04:40.477900 16452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:04:40.488158 16452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:40.489584 16452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:40.491024 16452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:40.515978 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:40.516000 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:40.516021 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:40.517009 16452 server_base.cc:1034] running on GCE node
I20260923 21:04:40.518096 16452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:40.519984 16452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:40.521306 16452 hybrid_clock.cc:648] HybridClock initialized: now 1790197480521238 us; error 65 us; skew 500 ppm
I20260923 21:04:40.521762 16452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:40 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:40.530210 16452 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:04:40.532094 16452 webserver.cc:460] Webserver started at http://127.0.97.62:44687/ using document root <none> and password file <none>
I20260923 21:04:40.533007 16452 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:40.533167 16452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:40.540274 16452 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 21:04:40.543753 16468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:40.545508 16452 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:04:40.545751 16452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "de2a29438d254ebbb1d0c880a341d7f4"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "6cee9f13ed384b30cb022e8b63f6516f"
server_key_iv: "05477a53338ce7a3672309ef1bafd714"
server_key_version: "encryptionkey@0"
I20260923 21:04:40.547001 16452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:40.579419 16452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:40.584507 16452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:40.584867 16452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:40.606580 16452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45325
I20260923 21:04:40.606602 16519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45325 every 8 connection(s)
I20260923 21:04:40.608376 16452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:04:40.608953 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16452
I20260923 21:04:40.610038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43539
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45325
--webserver_port=34331
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:04:40.616452 16520 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:04:40.620235 16520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Bootstrap starting.
I20260923 21:04:40.641256 16520 log.cc:824] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:40.654897 16520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:04:40.655588 16520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Bootstrap complete.
I20260923 21:04:40.665750 16520 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:40.666787 16520 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2a29438d254ebbb1d0c880a341d7f4, State: Initialized, Role: FOLLOWER
I20260923 21:04:40.667551 16520 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:40.667798 16520 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:04:40.667925 16520 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:04:40.668118 16520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 1 FOLLOWER]: Advancing to term 2
I20260923 21:04:40.672223 16520 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:40.672760 16520 leader_election.cc:302] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2a29438d254ebbb1d0c880a341d7f4; no voters:
I20260923 21:04:40.673477 16520 leader_election.cc:288] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 21:04:40.673620 16524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 21:04:40.674911 16524 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 LEADER]: Becoming Leader. State: Replica: de2a29438d254ebbb1d0c880a341d7f4, State: Running, Role: LEADER
I20260923 21:04:40.675592 16524 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:04:40.675916 16520 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:04:40.678203 16526 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2a29438d254ebbb1d0c880a341d7f4. Latest consensus state: current_term: 2 leader_uuid: "de2a29438d254ebbb1d0c880a341d7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } } }
I20260923 21:04:40.678973 16526 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:04:40.678614 16525 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de2a29438d254ebbb1d0c880a341d7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } } }
I20260923 21:04:40.679533 16525 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:04:40.692127 16530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:04:40.702265 16530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:04:40.704344 16530 catalog_manager.cc:1259] Loaded cluster ID: 68db847905e845e183c20c861ba72811
I20260923 21:04:40.704581 16530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:04:40.711084 16530 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:04:40.713306 16530 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Loaded TSK: 0
I20260923 21:04:40.726071 16541 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 21:04:40.750509 16530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Generated new TSK 1
I20260923 21:04:40.750998 16530 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 21:04:40.998473 16522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:40.998850 16522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:40.999018 16522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:41.006888 16522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:41.007076 16522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:41.007256 16522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:04:41.017853 16522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34331
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:41.019484 16522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:41.021026 16522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:41.036041 16550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:41.036301 16547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:41.036041 16548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:41.037963 16522 server_base.cc:1034] running on GCE node
I20260923 21:04:41.038721 16522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:41.039848 16522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:41.041119 16522 hybrid_clock.cc:648] HybridClock initialized: now 1790197481041050 us; error 55 us; skew 500 ppm
I20260923 21:04:41.041582 16522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:41 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:41.045617 16522 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:04:41.047590 16522 webserver.cc:460] Webserver started at http://127.0.97.1:34331/ using document root <none> and password file <none>
I20260923 21:04:41.048405 16522 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:41.048588 16522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:41.055424 16522 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 21:04:41.058970 16557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:41.060778 16522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:04:41.061002 16522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "5625f3fe57844f31bd14aff52ddb1a4f"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "adaeea6fbccca01c5b43fa83a3d386f2"
server_key_iv: "e1cec6a2f9eb8fd76cf33f433b2995ce"
server_key_version: "encryptionkey@0"
I20260923 21:04:41.062175 16522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:41.082517 16522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:41.087858 16522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:41.088213 16522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:41.089551 16522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:04:41.091169 16522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:04:41.091279 16522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:41.091454 16522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:04:41.091563 16522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:41.128517 16522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43539
I20260923 21:04:41.128609 16669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43539 every 8 connection(s)
I20260923 21:04:41.130158 16522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:04:41.136089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16522
I20260923 21:04:41.136790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45893
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45325
--webserver_port=41669
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:04:41 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:04:41.157768 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:41.133305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39521 (local address 127.0.97.62:45325)
0923 21:04:41.134201 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:41.134218 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:41.134638 (+ 420us) server_negotiation.cc:407] Connection header received
0923 21:04:41.137132 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:41.137153 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:41.137530 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:41.138044 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:41.139716 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.141000 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.142639 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.143979 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.144181 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:41.149291 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:41.149301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:41.149318 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:41.149363 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:41.152649 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:41.153687 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:41.153694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:41.153701 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:41.153857 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:41.154416 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:41.154420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:41.154422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:41.154918 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:41.155231 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:41.156051 (+ 820us) server_negotiation.cc:299] Negotiation successful
0923 21:04:41.156448 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":324,"threads_started":1}
I20260923 21:04:41.157768 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:41.133444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:39521)
0923 21:04:41.134449 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:41.134491 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:41.136390 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:41.138219 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:41.138228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:41.138911 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:04:41.139476 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:41.139495 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.141184 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.141188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:41.142495 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:41.142508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.144329 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.144342 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:41.144761 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:41.145675 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:04:41.145709 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:04:41.148851 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:04:41.152822 (+ 3971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:41.152834 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:41.152854 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:41.153535 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:41.153976 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:41.153984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:41.153990 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:41.154240 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:41.155159 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:04:41.155167 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:04:41.155539 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 21:04:41.155811 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 21:04:41.156383 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":371,"threads_started":1}
I20260923 21:04:41.161003 16670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45325
I20260923 21:04:41.161407 16670 heartbeater.cc:464] Registering TS with master...
I20260923 21:04:41.162395 16670 heartbeater.cc:511] Master 127.0.97.62:45325 requested a full tablet report, sending...
I20260923 21:04:41.164808 16485 ts_manager.cc:194] Registered new tserver with Master: 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539)
I20260923 21:04:41.166491 16485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39521
W20260923 21:04:41.495391 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:41.495718 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:41.495826 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:41.504096 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:41.504225 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:41.504344 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:04:41.514743 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41669
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:41.516095 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:41.517416 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:41.531635 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:41.531713 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:41.531729 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:41.533126 16674 server_base.cc:1034] running on GCE node
I20260923 21:04:41.533787 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:41.534857 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:41.536113 16674 hybrid_clock.cc:648] HybridClock initialized: now 1790197481536038 us; error 62 us; skew 500 ppm
I20260923 21:04:41.536420 16674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:41 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:41.540112 16674 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:04:41.541781 16674 webserver.cc:460] Webserver started at http://127.0.97.2:41669/ using document root <none> and password file <none>
I20260923 21:04:41.542773 16674 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:41.542946 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:41.549074 16674 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:04:41.552493 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:41.554108 16674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:04:41.554409 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "800274ba5ea14fd29c64f3cf09fbcbf0"
format_stamp: "Formatted at 2026-09-23 21:04:18 on dist-test-slave-8xn5"
server_key: "bf46ecbe8c126413270537ecc30164dd"
server_key_iv: "937f2f86cac08df835cba2d5dbb0acb7"
server_key_version: "encryptionkey@0"
I20260923 21:04:41.555510 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:41.580832 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:41.585626 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:41.585930 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:41.587000 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:04:41.588389 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:04:41.588487 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:41.588641 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:04:41.588742 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:41.624812 16674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45893
I20260923 21:04:41.624903 16801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45893 every 8 connection(s)
I20260923 21:04:41.626315 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:04:41.635622 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16674
I20260923 21:04:41.636494 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37535
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45325
--webserver_port=39087
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 21:04:41 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:04:41.649950 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:41.629082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38555 (local address 127.0.97.62:45325)
0923 21:04:41.629326 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:41.629331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:41.630753 (+ 1422us) server_negotiation.cc:407] Connection header received
0923 21:04:41.632790 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:41.632794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:41.632887 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:41.632978 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:41.634696 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.635409 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.638429 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.638919 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.639035 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:41.643747 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:41.643752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:41.643755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:41.643791 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:41.646824 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:41.647611 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:41.647616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:41.647618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:41.647742 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:41.648191 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:41.648201 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:41.648203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:41.648472 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:41.648633 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:41.649353 (+ 720us) server_negotiation.cc:299] Negotiation successful
0923 21:04:41.649592 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 21:04:41.651139 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:41.629556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:38555)
0923 21:04:41.630579 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:41.630633 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:41.632526 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:41.633201 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:41.633209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:41.633811 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:04:41.634513 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:41.634534 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.635606 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.635613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:41.638210 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:41.638233 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:41.639182 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:41.639192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:41.639552 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:41.640429 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:04:41.640470 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:04:41.643441 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:04:41.646976 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:41.646986 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:41.647009 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:41.647456 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:41.647861 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:41.647865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:41.647869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:41.648065 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:41.648607 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:04:41.648616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:04:41.648975 (+ 359us) client_negotiation.cc:769] Sending connection context
0923 21:04:41.649246 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 21:04:41.649814 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":406,"threads_started":1}
I20260923 21:04:41.652820 16802 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45325
I20260923 21:04:41.653142 16802 heartbeater.cc:464] Registering TS with master...
I20260923 21:04:41.653882 16802 heartbeater.cc:511] Master 127.0.97.62:45325 requested a full tablet report, sending...
I20260923 21:04:41.655536 16485 ts_manager.cc:194] Registered new tserver with Master: 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
I20260923 21:04:41.657015 16485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38555
W20260923 21:04:42.024008 16805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:04:42.024343 16805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:04:42.024471 16805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:04:42.033212 16805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:04:42.033342 16805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:04:42.033471 16805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:04:42.044903 16805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39087
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:04:42.046543 16805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:04:42.048283 16805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:04:42.063833 16810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:42.064220 16811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:04:42.064334 16813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:04:42.065435 16805 server_base.cc:1034] running on GCE node
I20260923 21:04:42.066154 16805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:04:42.067550 16805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:04:42.068809 16805 hybrid_clock.cc:648] HybridClock initialized: now 1790197482068723 us; error 96 us; skew 500 ppm
I20260923 21:04:42.069196 16805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:04:42 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:04:42.072626 16805 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:04:42.074163 16805 webserver.cc:460] Webserver started at http://127.0.97.3:39087/ using document root <none> and password file <none>
I20260923 21:04:42.074951 16805 fs_manager.cc:359] Metadata directory not provided
I20260923 21:04:42.075080 16805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:04:42.081168 16805 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:04:42.084386 16820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:42.085907 16805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:04:42.086134 16805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "5a13b2881b2048998fbbc0d63be1ac23"
format_stamp: "Formatted at 2026-09-23 21:04:19 on dist-test-slave-8xn5"
server_key: "a3681a113e6798b584b8a8e7e16c28b1"
server_key_iv: "a91d798f823b438be5d568982ea425b1"
server_key_version: "encryptionkey@0"
I20260923 21:04:42.087324 16805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:04:42.100448 16805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:04:42.105270 16805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:04:42.105580 16805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:04:42.106716 16805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:04:42.108220 16805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:04:42.108346 16805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:42.108489 16805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:04:42.108580 16805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:04:42.144181 16805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37535
I20260923 21:04:42.144270 16932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37535 every 8 connection(s)
I20260923 21:04:42.145622 16805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:04:42.155381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16805
Sep 23 21:04:42 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:04:42.169704 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.148840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60017 (local address 127.0.97.62:45325)
0923 21:04:42.149056 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.149061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.150689 (+ 1628us) server_negotiation.cc:407] Connection header received
0923 21:04:42.152708 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.152713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.152781 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.152878 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:42.154592 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.155339 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.157417 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.158078 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.158214 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.162433 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:42.162442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:42.162449 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:42.162493 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:42.166062 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.167090 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.167094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.167096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.167229 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.167754 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.167759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.167763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.168100 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:42.168220 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.169215 (+ 995us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.169410 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 21:04:42.170089 16670 heartbeater.cc:503] Master 127.0.97.62:45325 was elected leader, sending a full tablet report...
I20260923 21:04:42.171660 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.149337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:60017)
0923 21:04:42.150499 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.150554 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:42.152443 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:42.153203 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:42.153212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:42.153732 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:04:42.154322 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.154401 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.155538 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.155545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.157240 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.157251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.158307 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.158312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.158603 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.159316 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:04:42.159380 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:04:42.162181 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:04:42.166208 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:42.166215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:42.166229 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:42.166911 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:42.167347 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:04:42.167355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:04:42.167360 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:04:42.167588 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:04:42.168258 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:04:42.168268 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:04:42.168735 (+ 467us) client_negotiation.cc:769] Sending connection context
0923 21:04:42.169083 (+ 348us) client_negotiation.cc:240] Negotiation successful
0923 21:04:42.169642 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":652,"threads_started":1}
I20260923 21:04:42.173084 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.156429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.0.97.62:45325)
0923 21:04:42.156971 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.156977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.157310 (+ 333us) server_negotiation.cc:407] Connection header received
0923 21:04:42.157498 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.157502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.157564 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.157646 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:42.159700 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.160955 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.162509 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.163098 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.163231 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.166775 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:42.166781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:42.166785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:42.166853 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:42.169992 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.170891 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.170896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.170900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.171043 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.171437 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.171442 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.171445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.171839 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:42.172087 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.172487 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.172710 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":281,"threads_started":1}
I20260923 21:04:42.173211 16933 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45325
I20260923 21:04:42.173569 16933 heartbeater.cc:464] Registering TS with master...
I20260923 21:04:42.174307 16933 heartbeater.cc:511] Master 127.0.97.62:45325 requested a full tablet report, sending...
I20260923 21:04:42.175853 16485 ts_manager.cc:194] Registered new tserver with Master: 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535)
I20260923 21:04:42.176654 16485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60017
I20260923 21:04:42.188493 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:04:42.195888 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.189673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.0.97.62:45325)
0923 21:04:42.189933 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.189942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.190306 (+ 364us) server_negotiation.cc:407] Connection header received
0923 21:04:42.190610 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.190614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.190759 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.190924 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:04:42.191340 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.192271 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.193818 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.194243 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.194412 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.194980 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:04:42.195200 (+ 220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 21:04:42.195595 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41582: token expired
Metrics: {"server-negotiator.queue_time_us":85}
W20260923 21:04:42.196048 16936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41582: token expired
W20260923 21:04:42.196580 16404 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:45325)
I20260923 21:04:42.210768 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.197181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.97.62:45325)
0923 21:04:42.197384 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.197389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.197406 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:04:42.197605 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.197609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.197677 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.197776 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:42.199272 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.200016 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.201399 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.201814 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.201944 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.204610 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:42.204615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:42.204618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:42.204655 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:42.207507 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.208162 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.208167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.208169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.208298 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.208709 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.208713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.208715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.209153 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:42.209338 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.209878 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.210260 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 21:04:42.215570 16485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:04:42.224821 16485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:04:42.260370 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.237439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.0.97.1:43539)
0923 21:04:42.247796 (+ 10357us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.247813 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.249600 (+ 1787us) server_negotiation.cc:407] Connection header received
0923 21:04:42.249726 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.249737 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.250104 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.250395 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:42.251570 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.253246 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.256383 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.258943 (+ 2560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.259118 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.259447 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.259776 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.259962 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10058,"thread_start_us":10375,"threads_started":1}
I20260923 21:04:42.260704 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.236948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45664)
0923 21:04:42.248951 (+ 12003us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.249046 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:42.249332 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:42.250741 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:42.250750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:42.250786 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:42.251295 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.251312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.253446 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.253459 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.256203 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.256218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.259217 (+ 2999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.259223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.259450 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.259562 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 21:04:42.259711 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 21:04:42.259952 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11612,"mutex_wait_us":1,"thread_start_us":15414,"threads_started":1}
I20260923 21:04:42.261348 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.249866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.0.97.2:45893)
0923 21:04:42.251528 (+ 1662us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.251539 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.252808 (+ 1269us) server_negotiation.cc:407] Connection header received
0923 21:04:42.252952 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.252960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.253188 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.253393 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:42.254453 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.255969 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.258105 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.259678 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.259818 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.260044 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.260812 (+ 768us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.260996 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1477,"thread_start_us":1374,"threads_started":1}
I20260923 21:04:42.262084 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.237246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37535 (local address 127.0.0.1:52576)
0923 21:04:42.252328 (+ 15082us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.252351 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:42.252585 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:42.253319 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:42.253327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:42.253356 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:42.253823 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.253841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.256223 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.256230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.258574 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.258589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.261215 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.261221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.261387 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.261418 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 21:04:42.261507 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 21:04:42.261642 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14922,"mutex_wait_us":1,"thread_start_us":11110,"threads_started":1}
I20260923 21:04:42.262207 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.251629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45893 (local address 127.0.0.1:49264)
0923 21:04:42.252642 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.252685 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:42.252813 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:42.253654 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:42.253661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:42.253698 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:42.254213 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.254226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.256152 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.256163 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.257910 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.257926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.260244 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.260253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.260459 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.260521 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 21:04:42.260612 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 21:04:42.261730 (+ 1118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":26,"thread_start_us":358,"threads_started":1}
I20260923 21:04:42.263940 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.237619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.0.97.3:37535)
0923 21:04:42.251070 (+ 13451us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.251080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.252466 (+ 1386us) server_negotiation.cc:407] Connection header received
0923 21:04:42.252797 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.252806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.253066 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.253271 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:42.254015 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.256079 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.258773 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.260430 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.260595 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.261354 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.263341 (+ 1987us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.263549 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13259,"thread_start_us":13022,"threads_started":1}
I20260923 21:04:42.265796 16605 tablet_service.cc:1467] Processing CreateTablet for tablet a7f646e3e5b34ede8a22f6b493ce5de1 (DEFAULT_TABLE table=test-table [id=350d4f6b4d134a96b26f9fd70a2f70fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:04:42.265884 16737 tablet_service.cc:1467] Processing CreateTablet for tablet a7f646e3e5b34ede8a22f6b493ce5de1 (DEFAULT_TABLE table=test-table [id=350d4f6b4d134a96b26f9fd70a2f70fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:04:42.267421 16737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f646e3e5b34ede8a22f6b493ce5de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:04:42.267611 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f646e3e5b34ede8a22f6b493ce5de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:04:42.267818 16868 tablet_service.cc:1467] Processing CreateTablet for tablet a7f646e3e5b34ede8a22f6b493ce5de1 (DEFAULT_TABLE table=test-table [id=350d4f6b4d134a96b26f9fd70a2f70fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:04:42.269276 16868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f646e3e5b34ede8a22f6b493ce5de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:04:42.282131 16949 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Bootstrap starting.
I20260923 21:04:42.285037 16951 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Bootstrap starting.
I20260923 21:04:42.286724 16950 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Bootstrap starting.
I20260923 21:04:42.286942 16949 tablet_bootstrap.cc:654] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Neither blocks nor log segments found. Creating new log.
I20260923 21:04:42.288185 16949 log.cc:824] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:42.289304 16951 tablet_bootstrap.cc:654] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Neither blocks nor log segments found. Creating new log.
I20260923 21:04:42.290736 16950 tablet_bootstrap.cc:654] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Neither blocks nor log segments found. Creating new log.
I20260923 21:04:42.290923 16951 log.cc:824] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:42.292466 16950 log.cc:824] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Log is configured to *not* fsync() on all Append() calls
I20260923 21:04:42.292727 16949 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: No bootstrap required, opened a new log
I20260923 21:04:42.293063 16949 ts_tablet_manager.cc:1397] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 21:04:42.301028 16950 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: No bootstrap required, opened a new log
I20260923 21:04:42.301028 16951 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: No bootstrap required, opened a new log
I20260923 21:04:42.301348 16950 ts_tablet_manager.cc:1397] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260923 21:04:42.301348 16951 ts_tablet_manager.cc:1397] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260923 21:04:42.301090 16949 raft_consensus.cc:348] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.302567 16949 raft_consensus.cc:374] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:04:42.302743 16949 raft_consensus.cc:729] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5625f3fe57844f31bd14aff52ddb1a4f, State: Initialized, Role: FOLLOWER
I20260923 21:04:42.303750 16949 consensus_queue.cc:260] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.305563 16949 ts_tablet_manager.cc:1428] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 21:04:42.311733 16950 raft_consensus.cc:348] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.311962 16951 raft_consensus.cc:348] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.312335 16950 raft_consensus.cc:374] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:04:42.312466 16950 raft_consensus.cc:729] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a13b2881b2048998fbbc0d63be1ac23, State: Initialized, Role: FOLLOWER
I20260923 21:04:42.312491 16951 raft_consensus.cc:374] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:04:42.312614 16951 raft_consensus.cc:729] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 800274ba5ea14fd29c64f3cf09fbcbf0, State: Initialized, Role: FOLLOWER
I20260923 21:04:42.313186 16950 consensus_queue.cc:260] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.313218 16951 consensus_queue.cc:260] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.314419 16933 heartbeater.cc:503] Master 127.0.97.62:45325 was elected leader, sending a full tablet report...
I20260923 21:04:42.315182 16950 ts_tablet_manager.cc:1428] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 21:04:42.315204 16802 heartbeater.cc:503] Master 127.0.97.62:45325 was elected leader, sending a full tablet report...
I20260923 21:04:42.315928 16951 ts_tablet_manager.cc:1428] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260923 21:04:42.327630 16957 raft_consensus.cc:482] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:04:42.327939 16957 raft_consensus.cc:504] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.329726 16957 leader_election.cc:288] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539), 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535)
I20260923 21:04:42.337530 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.330117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52805 (local address 127.0.97.1:43539)
0923 21:04:42.330449 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.330459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.331421 (+ 962us) server_negotiation.cc:407] Connection header received
0923 21:04:42.331643 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.331652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.331747 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.331844 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:42.332853 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.333619 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.335546 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.336388 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.336572 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.336691 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.337128 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.337278 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260923 21:04:42.337723 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.329971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.97.2:52805)
0923 21:04:42.331262 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.331285 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:42.331480 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:42.332161 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:42.332165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:42.332203 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:42.332710 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.332719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.333748 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.333756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.335377 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.335389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.336663 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.336670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.336878 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.336995 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 21:04:42.337101 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 21:04:42.337259 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1117,"mutex_wait_us":4,"thread_start_us":884,"threads_started":1}
I20260923 21:04:42.337867 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.330485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42167 (local address 127.0.97.3:37535)
0923 21:04:42.330735 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.330744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.331712 (+ 968us) server_negotiation.cc:407] Connection header received
0923 21:04:42.331864 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.331867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.331927 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.331998 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:04:42.332835 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.333665 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.335723 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.336611 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.336779 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.336897 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.337469 (+ 572us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.337604 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 21:04:42.338402 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.330121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37535 (local address 127.0.97.2:42167)
0923 21:04:42.331595 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.331614 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:04:42.331750 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:04:42.332101 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:04:42.332109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:04:42.332139 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:04:42.332637 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.332650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.333817 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.333823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.335552 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:04:42.335564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.336849 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.336856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:04:42.337295 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.337321 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 21:04:42.337439 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 21:04:42.337598 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1228,"mutex_wait_us":139,"spinlock_wait_cycles":27904,"thread_start_us":1057,"threads_started":1}
I20260923 21:04:42.338899 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f646e3e5b34ede8a22f6b493ce5de1" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" is_pre_election: true
I20260923 21:04:42.339555 16625 raft_consensus.cc:2454] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 0.
I20260923 21:04:42.339499 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f646e3e5b34ede8a22f6b493ce5de1" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a13b2881b2048998fbbc0d63be1ac23" is_pre_election: true
I20260923 21:04:42.340088 16888 raft_consensus.cc:2454] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 0.
I20260923 21:04:42.340308 16691 leader_election.cc:302] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f, 800274ba5ea14fd29c64f3cf09fbcbf0; no voters:
I20260923 21:04:42.340757 16957 raft_consensus.cc:2781] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:04:42.340914 16957 raft_consensus.cc:482] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:04:42.341013 16957 raft_consensus.cc:3037] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:04:42.343307 16957 raft_consensus.cc:504] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.344053 16957 leader_election.cc:288] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 election: Requested vote from peers 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539), 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535)
I20260923 21:04:42.344637 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f646e3e5b34ede8a22f6b493ce5de1" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5625f3fe57844f31bd14aff52ddb1a4f"
I20260923 21:04:42.344746 16888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f646e3e5b34ede8a22f6b493ce5de1" candidate_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a13b2881b2048998fbbc0d63be1ac23"
I20260923 21:04:42.344837 16625 raft_consensus.cc:3037] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:04:42.345055 16888 raft_consensus.cc:3037] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:04:42.347209 16625 raft_consensus.cc:2454] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 1.
I20260923 21:04:42.347457 16888 raft_consensus.cc:2454] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 800274ba5ea14fd29c64f3cf09fbcbf0 in term 1.
I20260923 21:04:42.347667 16691 leader_election.cc:302] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f, 800274ba5ea14fd29c64f3cf09fbcbf0; no voters:
I20260923 21:04:42.348104 16957 raft_consensus.cc:2781] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:04:42.348558 16957 raft_consensus.cc:686] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 LEADER]: Becoming Leader. State: Replica: 800274ba5ea14fd29c64f3cf09fbcbf0, State: Running, Role: LEADER
I20260923 21:04:42.349131 16957 consensus_queue.cc:237] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:04:42.354000 16483 catalog_manager.cc:5495] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } health_report { overall_health: HEALTHY } } }
I20260923 21:04:42.376047 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.370366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.0.97.2:45893)
0923 21:04:42.370594 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.370600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.370707 (+ 107us) server_negotiation.cc:407] Connection header received
0923 21:04:42.370969 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.371005 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.371097 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.371261 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:04:42.371631 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.372360 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.373677 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.374098 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.374235 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.374837 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:04:42.375357 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:04:42.375520 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.375600 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.375682 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 21:04:42.378469 16803 tablet.cc:2369] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:04:42.382972 16671 tablet.cc:2369] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:04:42.385571 16625 raft_consensus.cc:1261] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Refusing update from remote peer 800274ba5ea14fd29c64f3cf09fbcbf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:04:42.385571 16888 raft_consensus.cc:1261] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Refusing update from remote peer 800274ba5ea14fd29c64f3cf09fbcbf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:04:42.386675 16960 consensus_queue.cc:1035] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:04:42.387037 16957 consensus_queue.cc:1035] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 21:04:42.397351 16934 tablet.cc:2369] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:04:42.415763 16484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:04:42.416090 16484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:04:42.419673 16484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Sending DeleteTablet for 3 replicas of tablet a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:04:42.420953 16868 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f646e3e5b34ede8a22f6b493ce5de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:04:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52576
I20260923 21:04:42.420933 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f646e3e5b34ede8a22f6b493ce5de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:04:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45664
I20260923 21:04:42.421114 16737 tablet_service.cc:1514] Processing DeleteTablet for tablet a7f646e3e5b34ede8a22f6b493ce5de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:04:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49264
I20260923 21:04:42.421502 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 16522
I20260923 21:04:42.423448 16972 tablet_replica.cc:330] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: stopping tablet replica
I20260923 21:04:42.423985 16972 raft_consensus.cc:2229] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:04:42.424629 16972 raft_consensus.cc:2258] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:04:42.427016 16972 ts_tablet_manager.cc:1905] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:04:42.437685 16973 tablet_replica.cc:330] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: stopping tablet replica
I20260923 21:04:42.438199 16973 raft_consensus.cc:2229] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:04:42.438973 16973 raft_consensus.cc:2258] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:04:42.439316 16972 ts_tablet_manager.cc:1918] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:04:42.440028 16971 tablet_replica.cc:330] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: stopping tablet replica
I20260923 21:04:42.440461 16971 raft_consensus.cc:2229] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:04:42.440886 16972 log.cc:1196] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:04:42.441063 16971 raft_consensus.cc:2258] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:04:42.442041 16972 ts_tablet_manager.cc:1939] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5a13b2881b2048998fbbc0d63be1ac23: Deleting consensus metadata
I20260923 21:04:42.443008 16973 ts_tablet_manager.cc:1905] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:04:42.443835 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.422622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.0.97.1:43539)
0923 21:04:42.422884 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.422894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.423046 (+ 152us) server_negotiation.cc:407] Connection header received
0923 21:04:42.423270 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.423278 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.423414 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.423547 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:42.425399 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.427041 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.429005 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.429977 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.430144 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.433770 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:42.433796 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:42.433824 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:42.433893 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:42.439544 (+ 5651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.440878 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.440890 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.440894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.441251 (+ 357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.441720 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.441725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.441729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.442207 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:42.442578 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.443182 (+ 604us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.443450 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 21:04:42.444954 16665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:42.445348 16469 catalog_manager.cc:4841] TS 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535): tablet a7f646e3e5b34ede8a22f6b493ce5de1 (table test-table [id=350d4f6b4d134a96b26f9fd70a2f70fb]) successfully deleted
I20260923 21:04:42.449393 16973 ts_tablet_manager.cc:1918] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:04:42.449607 16973 log.cc:1196] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:04:42.450282 16973 ts_tablet_manager.cc:1939] T a7f646e3e5b34ede8a22f6b493ce5de1 P 800274ba5ea14fd29c64f3cf09fbcbf0: Deleting consensus metadata
I20260923 21:04:42.452400 16469 catalog_manager.cc:4841] TS 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893): tablet a7f646e3e5b34ede8a22f6b493ce5de1 (table test-table [id=350d4f6b4d134a96b26f9fd70a2f70fb]) successfully deleted
I20260923 21:04:42.543210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 16522
I20260923 21:04:42.561620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 16674
W20260923 21:04:42.561563 16470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43539 (error 108)
I20260923 21:04:42.564091 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.563568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45684)
0923 21:04:42.563780 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.563916 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 21:04:42.564358 16470 catalog_manager.cc:4584] TS 5625f3fe57844f31bd14aff52ddb1a4f (127.0.97.1:43539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7f646e3e5b34ede8a22f6b493ce5de1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
I20260923 21:04:42.577913 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.562531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.0.97.2:45893)
0923 21:04:42.562718 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.562725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.563358 (+ 633us) server_negotiation.cc:407] Connection header received
0923 21:04:42.563461 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.563467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.563552 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.563661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:42.565267 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.566087 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.567601 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.568033 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.568182 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.571090 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:42.571105 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:42.571120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:42.571157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:42.574812 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.575571 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.575583 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.575592 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.575739 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.576207 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.576216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.576219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.576647 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:42.576979 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.577261 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.577496 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:04:42.578816 16797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:42.618216 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.617585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45700)
0923 21:04:42.617854 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.618009 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 21:04:42.619562 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.619130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45710)
0923 21:04:42.619306 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.619418 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 21:04:42.680382 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.679847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45716)
0923 21:04:42.680127 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.680223 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260923 21:04:42.681522 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.681152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45718)
0923 21:04:42.681319 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.681403 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 21:04:42.695223 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 16674
I20260923 21:04:42.714926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 16805
I20260923 21:04:42.729910 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.715737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.0.97.3:37535)
0923 21:04:42.716169 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 21:04:42.716173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:04:42.716188 (+ 15us) server_negotiation.cc:407] Connection header received
0923 21:04:42.716240 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:04:42.716243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:04:42.716300 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:04:42.716378 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:04:42.717710 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.718544 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.720022 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:04:42.720395 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:04:42.720495 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:04:42.723609 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:04:42.723626 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:04:42.723643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:04:42.723700 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:04:42.726959 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.727831 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.727840 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.727842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.728003 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:04:42.728412 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:04:42.728415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:04:42.728417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:04:42.728810 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:04:42.729023 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 21:04:42.729346 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 21:04:42.729548 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260923 21:04:42.730782 16928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:42.762707 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.761916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45724)
0923 21:04:42.762162 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.762265 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 21:04:42.764030 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.763565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45734)
0923 21:04:42.763800 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.763883 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260923 21:04:42.822285 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 16805
I20260923 21:04:42.839403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 16452
I20260923 21:04:42.840657 16515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:04:42.960443 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:04:42.959918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43539 (local address 127.0.0.1:45748)
0923 21:04:42.960082 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 21:04:42.960214 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 21:04:42.960428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 16452
I20260923 21:05:02.982966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45325
--webserver_interface=127.0.97.62
--webserver_port=44687
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:03.336848 16979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:03.337208 16979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:03.337324 16979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:03.344980 16979 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 21:05:03.345108 16979 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 21:05:03.345147 16979 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 21:05:03.345248 16979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:03.345355 16979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:03.345476 16979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:03.345587 16979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:03.345672 16979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:03.356730 16979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:03.358269 16979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:03.359700 16979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:03.372627 16987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:03.372737 16984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:03.372980 16979 server_base.cc:1034] running on GCE node
W20260923 21:05:03.372663 16985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:03.374241 16979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:03.375968 16979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:03.377238 16979 hybrid_clock.cc:648] HybridClock initialized: now 1790197503377146 us; error 48 us; skew 500 ppm
I20260923 21:05:03.377661 16979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:03 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:03.381238 16979 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:05:03.382848 16979 webserver.cc:460] Webserver started at http://127.0.97.62:44687/ using document root <none> and password file <none>
I20260923 21:05:03.383560 16979 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:03.383652 16979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:03.389696 16979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 21:05:03.393078 16994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:03.394833 16979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 21:05:03.395040 16979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "de2a29438d254ebbb1d0c880a341d7f4"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "6cee9f13ed384b30cb022e8b63f6516f"
server_key_iv: "05477a53338ce7a3672309ef1bafd714"
server_key_version: "encryptionkey@0"
I20260923 21:05:03.396239 16979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:03.418905 16979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:03.424335 16979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:03.424693 16979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:03.447233 16979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45325
I20260923 21:05:03.447507 17045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45325 every 8 connection(s)
I20260923 21:05:03.448887 16979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:03.456183 17046 sys_catalog.cc:263] Verifying existing consensus state
I20260923 21:05:03.458252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 16979
I20260923 21:05:03.459299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43539
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45325
--webserver_port=34331
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:05:03.460584 17046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Bootstrap starting.
I20260923 21:05:03.483894 17046 log.cc:824] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:03.503319 17046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:05:03.503993 17046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Bootstrap complete.
I20260923 21:05:03.514269 17046 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:05:03.515399 17046 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2a29438d254ebbb1d0c880a341d7f4, State: Initialized, Role: FOLLOWER
I20260923 21:05:03.516127 17046 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:05:03.516402 17046 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:03.516532 17046 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:03.516752 17046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 2 FOLLOWER]: Advancing to term 3
I20260923 21:05:03.521178 17046 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:05:03.521766 17046 leader_election.cc:302] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2a29438d254ebbb1d0c880a341d7f4; no voters:
I20260923 21:05:03.522604 17046 leader_election.cc:288] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 21:05:03.522773 17050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 21:05:03.523861 17050 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [term 3 LEADER]: Becoming Leader. State: Replica: de2a29438d254ebbb1d0c880a341d7f4, State: Running, Role: LEADER
I20260923 21:05:03.524545 17050 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } }
I20260923 21:05:03.525014 17046 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:03.527201 17052 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2a29438d254ebbb1d0c880a341d7f4. Latest consensus state: current_term: 3 leader_uuid: "de2a29438d254ebbb1d0c880a341d7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } } }
I20260923 21:05:03.527732 17052 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:03.528141 17051 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "de2a29438d254ebbb1d0c880a341d7f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2a29438d254ebbb1d0c880a341d7f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45325 } } }
I20260923 21:05:03.528620 17051 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:03.534368 17057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:03.543537 17057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:03.545156 17057 catalog_manager.cc:1259] Loaded cluster ID: 68db847905e845e183c20c861ba72811
I20260923 21:05:03.545272 17057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:03.549726 17057 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:03.551553 17057 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Loaded TSK: 0
I20260923 21:05:03.551713 17057 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de2a29438d254ebbb1d0c880a341d7f4: Loaded TSK: 1
F20260923 21:05:03.552966 17057 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcb209f39ad google::LogMessage::Fail() at ??:0
@ 0x7fcb209f688a google::LogMessage::SendToLog() at ??:0
@ 0x7fcb209f34de google::LogMessage::Flush() at ??:0
@ 0x7fcb209f4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcb247577d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcb2474673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fcb364221c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcb36421783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcb364ad2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fcb364ad0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcb364f75e9 std::function<>::operator()() at ??:0
@ 0x7fcb36425cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcb36423c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcb364ab610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcb364ab439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcb2363f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcb2364291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcb23642799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcb2361deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcb268f76db start_thread at ??:0
@ 0x7fcb1f88d61f clone at ??:0
W20260923 21:05:03.880620 17048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:03.880935 17048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:03.881038 17048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:03.888579 17048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:03.888716 17048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:03.888845 17048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:05:03.899312 17048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34331
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:03.900658 17048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:03.902022 17048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:03.916086 17094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:03.916150 17093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:03.916150 17096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:03.917076 17048 server_base.cc:1034] running on GCE node
I20260923 21:05:03.917701 17048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:03.918725 17048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:03.919927 17048 hybrid_clock.cc:648] HybridClock initialized: now 1790197503919882 us; error 38 us; skew 500 ppm
I20260923 21:05:03.920252 17048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:03 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:03.923581 17048 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:05:03.925087 17048 webserver.cc:460] Webserver started at http://127.0.97.1:34331/ using document root <none> and password file <none>
I20260923 21:05:03.925815 17048 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:03.925966 17048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:03.931731 17048 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:05:03.935010 17103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:03.936581 17048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:05:03.936833 17048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "5625f3fe57844f31bd14aff52ddb1a4f"
format_stamp: "Formatted at 2026-09-23 21:04:17 on dist-test-slave-8xn5"
server_key: "adaeea6fbccca01c5b43fa83a3d386f2"
server_key_iv: "e1cec6a2f9eb8fd76cf33f433b2995ce"
server_key_version: "encryptionkey@0"
I20260923 21:05:03.937949 17048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:03.956926 17048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:03.961616 17048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:03.961915 17048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:03.963084 17048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:03.965065 17110 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260923 21:05:03.971658 17048 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 21:05:03.971841 17048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260923 21:05:03.972010 17048 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 21:05:03.975517 17048 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 21:05:03.975641 17048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260923 21:05:03.976070 17110 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Bootstrap starting.
I20260923 21:05:03.986953 17110 log.cc:824] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:04.003764 17110 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 21:05:04.004776 17110 tablet_bootstrap.cc:492] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Bootstrap complete.
I20260923 21:05:04.005366 17110 ts_tablet_manager.cc:1397] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.000s
I20260923 21:05:04.016742 17110 raft_consensus.cc:348] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:05:04.018077 17110 raft_consensus.cc:729] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5625f3fe57844f31bd14aff52ddb1a4f, State: Initialized, Role: FOLLOWER
I20260923 21:05:04.018687 17048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43539
I20260923 21:05:04.018729 17217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43539 every 8 connection(s)
I20260923 21:05:04.019052 17110 consensus_queue.cc:260] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:05:04.020358 17048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:05:04.020663 17110 ts_tablet_manager.cc:1428] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260923 21:05:04.025168 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:04.023340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:58145)
0923 21:05:04.024124 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:04.024317 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":343,"threads_started":1}
W20260923 21:05:04.025930 17219 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45325 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45325: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
I20260923 21:05:04.026881 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:04.026553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:40989)
0923 21:05:04.026702 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:04.026774 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 21:05:04.027715 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:04.027402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:59727)
0923 21:05:04.027531 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:04.027600 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 21:05:04.028014 17219 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 21:05:04.028915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 17048
I20260923 21:05:04.029487 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45893
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45325
--webserver_port=41669
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:04.396021 17222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:04.396394 17222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:04.396576 17222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:04.404456 17222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:04.404628 17222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:04.404816 17222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:05:04.415388 17222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41669
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:04.416842 17222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:04.418526 17222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:04.432962 17230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:04.432930 17227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:04.433177 17228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:04.434388 17222 server_base.cc:1034] running on GCE node
I20260923 21:05:04.435097 17222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:04.436126 17222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:04.437337 17222 hybrid_clock.cc:648] HybridClock initialized: now 1790197504437297 us; error 45 us; skew 500 ppm
I20260923 21:05:04.437736 17222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:04 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:04.441905 17222 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:05:04.443948 17222 webserver.cc:460] Webserver started at http://127.0.97.2:41669/ using document root <none> and password file <none>
I20260923 21:05:04.444710 17222 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:04.444857 17222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:04.451709 17222 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:05:04.455026 17237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:04.456655 17222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:05:04.456881 17222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "800274ba5ea14fd29c64f3cf09fbcbf0"
format_stamp: "Formatted at 2026-09-23 21:04:18 on dist-test-slave-8xn5"
server_key: "bf46ecbe8c126413270537ecc30164dd"
server_key_iv: "937f2f86cac08df835cba2d5dbb0acb7"
server_key_version: "encryptionkey@0"
I20260923 21:05:04.458088 17222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:04.513811 17222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:04.518891 17222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:04.519246 17222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:04.520689 17222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:04.522579 17222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:04.522737 17222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:04.522887 17222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:04.522979 17222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:04.561736 17222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45893
I20260923 21:05:04.561836 17349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45893 every 8 connection(s)
I20260923 21:05:04.563726 17222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:05:04.567239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 17222
I20260923 21:05:04.567885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37535
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45325
--webserver_port=39087
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:05:04.569680 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:04.567237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:48721)
0923 21:05:04.568310 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:04.568612 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":453,"threads_started":1}
W20260923 21:05:04.570501 17350 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45325 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45325: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
I20260923 21:05:04.571520 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:04.571148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:59069)
0923 21:05:04.571324 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:04.571400 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 21:05:04.572551 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:04.572147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:49571)
0923 21:05:04.572338 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:04.572431 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 21:05:04.572834 17350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 21:05:04.906904 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:04.907191 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:04.907351 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:04.915091 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:04.915208 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:04.915388 17353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:05:04.925381 17353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39087
--tserver_master_addrs=127.0.97.62:45325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:04.926807 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:04.928740 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:04.942979 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:04.943013 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:04.943045 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:04.944110 17353 server_base.cc:1034] running on GCE node
I20260923 21:05:04.944798 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:04.945864 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:04.947120 17353 hybrid_clock.cc:648] HybridClock initialized: now 1790197504947069 us; error 38 us; skew 500 ppm
I20260923 21:05:04.947479 17353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:04 dist-test-slave-8xn5 krb5kdc[15897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:04.951007 17353 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:05:04.952610 17353 webserver.cc:460] Webserver started at http://127.0.97.3:39087/ using document root <none> and password file <none>
I20260923 21:05:04.953375 17353 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:04.953541 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:04.959378 17353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:05:04.962400 17368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:04.964001 17353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 21:05:04.964223 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "5a13b2881b2048998fbbc0d63be1ac23"
format_stamp: "Formatted at 2026-09-23 21:04:19 on dist-test-slave-8xn5"
server_key: "a3681a113e6798b584b8a8e7e16c28b1"
server_key_iv: "a91d798f823b438be5d568982ea425b1"
server_key_version: "encryptionkey@0"
I20260923 21:05:04.965271 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:04.979512 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:04.983906 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:04.984213 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:04.985237 17353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:04.986707 17353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:04.986851 17353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:04.986986 17353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:04.987076 17353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:05.022543 17353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37535
I20260923 21:05:05.022583 17480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37535 every 8 connection(s)
I20260923 21:05:05.024112 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:05:05.029266 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.028574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:37925)
0923 21:05:05.028989 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:05.029062 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260923 21:05:05.029497 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.027273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:57695)
0923 21:05:05.028122 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:05.028362 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":350,"threads_started":1}
W20260923 21:05:05.030135 17481 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45325 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45325: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
I20260923 21:05:05.031020 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.030712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:37081)
0923 21:05:05.030858 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:05.030924 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 21:05:05.031857 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.031563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:52349)
0923 21:05:05.031677 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:05.031756 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 21:05:05.032117 17481 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 21:05:05.033385 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 17353
I20260923 21:05:05.574240 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.573411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:59827)
0923 21:05:05.573896 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:05.574005 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":208,"threads_started":1}
I20260923 21:05:05.720096 17487 raft_consensus.cc:482] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:05.720482 17487 raft_consensus.cc:504] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:05:05.722091 17487 leader_election.cc:288] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
Sep 23 21:05:05 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 23 21:05:05 dist-test-slave-8xn5 krb5kdc[15897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:05:05.746955 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.722522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37535 (local address 127.0.97.1:60337)
0923 21:05:05.723198 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:05.723240 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:05.723893 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:05.728558 (+ 4665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:05.728569 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:05.729444 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:05.730284 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:05.730296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.731826 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.731838 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:05.733194 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:05.733206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.734280 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.734287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:05.734551 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:05.735455 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:05.735480 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:05.738663 (+ 3183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:05.742389 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:05.742421 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:05.742440 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:05.743242 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:05.743674 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:05.743677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:05.743679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:05.743864 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:05.744937 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:05.744945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:05.745501 (+ 556us) client_negotiation.cc:769] Sending connection context
0923 21:05:05.745825 (+ 324us) client_negotiation.cc:240] Negotiation successful
0923 21:05:05.746492 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":162,"spinlock_wait_cycles":37888,"thread_start_us":294,"threads_started":1}
I20260923 21:05:05.747002 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.722719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60337 (local address 127.0.97.3:37535)
0923 21:05:05.725929 (+ 3210us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:05.725953 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:05.726007 (+ 54us) server_negotiation.cc:407] Connection header received
0923 21:05:05.727658 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:05.727667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:05.728077 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:05.728427 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:05.730528 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.731656 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.733347 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.734015 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.734244 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:05.738986 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:05.738996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:05.739012 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:05.739058 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:05.742203 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:05.743393 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:05.743400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:05.743406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:05.743569 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:05.744023 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:05.744027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:05.744029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:05.744765 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:05.744975 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:05.746090 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0923 21:05:05.746563 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2963,"thread_start_us":2626,"threads_started":1}
I20260923 21:05:05.747032 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.723360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45893 (local address 127.0.97.1:41043)
0923 21:05:05.726395 (+ 3035us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:05.726410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:05.726557 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:05.728814 (+ 2257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:05.728818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:05.729483 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:05.730143 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:05.730162 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.731750 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.731754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:05.733200 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:05.733212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.734227 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.734234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:05.734545 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:05.735373 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:05.735396 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:05.738725 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:05.742460 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:05.742466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:05.742469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:05.743242 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:05.743972 (+ 730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:05.743977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:05.743982 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:05.744118 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:05.744943 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:05.744947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:05.745501 (+ 554us) client_negotiation.cc:769] Sending connection context
0923 21:05:05.745825 (+ 324us) client_negotiation.cc:240] Negotiation successful
0923 21:05:05.746602 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2884,"mutex_wait_us":456,"spinlock_wait_cycles":6528,"thread_start_us":2714,"threads_started":1}
I20260923 21:05:05.747921 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:05.723887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41043 (local address 127.0.97.2:45893)
0923 21:05:05.726077 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:05.726095 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:05.726514 (+ 419us) server_negotiation.cc:407] Connection header received
0923 21:05:05.728199 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:05.728207 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:05.728540 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:05.728848 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:05.730391 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.731546 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.733353 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:05.734014 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:05.734220 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:05.738998 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:05.739005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:05.739018 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:05.739060 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:05.742292 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:05.743712 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:05.743717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:05.743723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:05.743863 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:05.744227 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:05.744231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:05.744234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:05.744825 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:05.744989 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:05.746090 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0923 21:05:05.746563 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1848,"threads_started":1}
W20260923 21:05:05.750934 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
W20260923 21:05:05.751345 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:05:05.751515 17104 leader_election.cc:302] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f; no voters: 5a13b2881b2048998fbbc0d63be1ac23, 800274ba5ea14fd29c64f3cf09fbcbf0
I20260923 21:05:05.751969 17487 raft_consensus.cc:2726] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 21:05:06.030758 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:06.030161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:50911)
0923 21:05:06.030463 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:06.030586 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260923 21:05:06.033459 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:06.032618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:40287)
0923 21:05:06.033060 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:06.033188 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":217,"threads_started":1}
I20260923 21:05:06.576149 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:06.575294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:47735)
0923 21:05:06.575825 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:06.575949 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":322,"threads_started":1}
I20260923 21:05:07.032513 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:07.031602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:34333)
0923 21:05:07.032182 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:07.032297 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":274,"threads_started":1}
I20260923 21:05:07.035039 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:07.034269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:60029)
0923 21:05:07.034820 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:07.034895 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":353,"threads_started":1}
I20260923 21:05:07.403836 17496 raft_consensus.cc:482] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:07.404070 17496 raft_consensus.cc:504] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:05:07.404755 17496 leader_election.cc:288] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
W20260923 21:05:07.405854 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
W20260923 21:05:07.406101 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:05:07.406214 17104 leader_election.cc:302] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f; no voters: 5a13b2881b2048998fbbc0d63be1ac23, 800274ba5ea14fd29c64f3cf09fbcbf0
I20260923 21:05:07.406507 17496 raft_consensus.cc:2726] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 21:05:07.577950 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:07.577177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:46473)
0923 21:05:07.577684 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:07.577776 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
I20260923 21:05:08.034595 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:08.033527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:53135)
0923 21:05:08.034117 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:08.034212 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
I20260923 21:05:08.036631 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:08.035877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:35281)
0923 21:05:08.036412 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:08.036485 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":286,"threads_started":1}
I20260923 21:05:08.579905 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:08.579062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:39703)
0923 21:05:08.579650 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:08.579729 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":345,"threads_started":1}
I20260923 21:05:09.036526 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:09.035739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:43329)
0923 21:05:09.036252 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:09.036347 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":242,"threads_started":1}
I20260923 21:05:09.038188 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:09.037439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:44173)
0923 21:05:09.037842 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:09.037953 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":207,"threads_started":1}
I20260923 21:05:09.284174 17503 raft_consensus.cc:482] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:09.284365 17503 raft_consensus.cc:504] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:05:09.285171 17503 leader_election.cc:288] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
W20260923 21:05:09.286260 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
W20260923 21:05:09.286566 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:05:09.286720 17104 leader_election.cc:302] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f; no voters: 5a13b2881b2048998fbbc0d63be1ac23, 800274ba5ea14fd29c64f3cf09fbcbf0
I20260923 21:05:09.287154 17503 raft_consensus.cc:2726] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 21:05:09.581702 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:09.580932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:35645)
0923 21:05:09.581422 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:09.581533 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":259,"threads_started":1}
I20260923 21:05:10.038928 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:10.037678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:59617)
0923 21:05:10.038441 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:10.038638 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":257,"threads_started":1}
I20260923 21:05:10.040220 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:10.039408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:55751)
0923 21:05:10.039926 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:10.040016 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":312,"threads_started":1}
I20260923 21:05:10.583647 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:10.582802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:36055)
0923 21:05:10.583300 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:10.583401 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":297,"threads_started":1}
I20260923 21:05:11.040930 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:11.040047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:41327)
0923 21:05:11.040550 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:11.040639 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":263,"threads_started":1}
I20260923 21:05:11.042110 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:11.041161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:38135)
0923 21:05:11.041831 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:11.041915 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":430,"threads_started":1}
I20260923 21:05:11.585515 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:11.584665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:35841)
0923 21:05:11.585223 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:11.585313 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":245,"threads_started":1}
I20260923 21:05:12.042971 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:12.042099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:57909)
0923 21:05:12.042640 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:12.042761 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":317,"threads_started":1}
I20260923 21:05:12.044128 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:12.043285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:58461)
0923 21:05:12.043881 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:12.043963 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":318,"threads_started":1}
I20260923 21:05:12.587328 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:12.586567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:53807)
0923 21:05:12.587054 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:12.587153 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260923 21:05:12.723806 17514 raft_consensus.cc:482] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:12.724005 17514 raft_consensus.cc:504] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:05:12.724570 17514 leader_election.cc:288] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
W20260923 21:05:12.725603 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
W20260923 21:05:12.725852 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:05:12.725955 17104 leader_election.cc:302] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f; no voters: 5a13b2881b2048998fbbc0d63be1ac23, 800274ba5ea14fd29c64f3cf09fbcbf0
I20260923 21:05:12.726248 17514 raft_consensus.cc:2726] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 21:05:13.045291 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:13.044121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:37251)
0923 21:05:13.044833 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:13.044954 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":355,"threads_started":1}
I20260923 21:05:13.045954 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:13.045057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:41799)
0923 21:05:13.045630 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:13.045741 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":272,"threads_started":1}
I20260923 21:05:13.589284 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:13.588382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:39303)
0923 21:05:13.588967 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:13.589071 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":292,"threads_started":1}
I20260923 21:05:14.047441 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:14.046490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:33411)
0923 21:05:14.047089 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:14.047199 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260923 21:05:14.047855 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:14.047077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:59311)
0923 21:05:14.047568 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:14.047693 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":223,"threads_started":1}
I20260923 21:05:14.591136 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:14.590392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:42707)
0923 21:05:14.590865 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:14.590958 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":230,"threads_started":1}
I20260923 21:05:15.049625 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:15.048817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:47909)
0923 21:05:15.049372 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:15.049454 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":345,"threads_started":1}
I20260923 21:05:15.049623 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:15.048588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:58479)
0923 21:05:15.049336 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:15.049430 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":496,"threads_started":1}
I20260923 21:05:15.592898 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:15.592133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:60875)
0923 21:05:15.592626 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:15.592724 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
I20260923 21:05:16.051856 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:16.050826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:58015)
0923 21:05:16.051562 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:16.051659 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":349,"threads_started":1}
I20260923 21:05:16.051882 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:16.050992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:53005)
0923 21:05:16.051602 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:16.051704 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":356,"threads_started":1}
I20260923 21:05:16.595198 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:16.594186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:42665)
0923 21:05:16.594897 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:16.595006 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":347,"threads_started":1}
I20260923 21:05:17.053926 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:17.053069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:33857)
0923 21:05:17.053625 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:17.053712 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":315,"threads_started":1}
I20260923 21:05:17.054473 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:17.053073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:33407)
0923 21:05:17.054080 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:17.054212 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":751,"threads_started":1}
I20260923 21:05:17.597095 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:17.596236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:37189)
0923 21:05:17.596753 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:17.596918 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":271,"threads_started":1}
I20260923 21:05:18.056046 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:18.055176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:41981)
0923 21:05:18.055756 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:18.055849 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":329,"threads_started":1}
I20260923 21:05:18.056370 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:18.055526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:39139)
0923 21:05:18.056085 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:18.056165 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":324,"threads_started":1}
I20260923 21:05:18.599009 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:18.598086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:51945)
0923 21:05:18.598673 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:18.598786 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":332,"threads_started":1}
I20260923 21:05:19.058454 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:19.057594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:58827)
0923 21:05:19.058144 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:19.058218 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":309,"threads_started":1}
I20260923 21:05:19.058547 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:19.057484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:52029)
0923 21:05:19.058124 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:19.058234 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":339,"threads_started":1}
I20260923 21:05:19.438565 17535 raft_consensus.cc:482] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:19.438866 17535 raft_consensus.cc:504] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5625f3fe57844f31bd14aff52ddb1a4f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43539 } } peers { permanent_uuid: "5a13b2881b2048998fbbc0d63be1ac23" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37535 } } peers { permanent_uuid: "800274ba5ea14fd29c64f3cf09fbcbf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45893 } }
I20260923 21:05:19.439594 17535 leader_election.cc:288] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535), 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893)
W20260923 21:05:19.440750 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5a13b2881b2048998fbbc0d63be1ac23 (127.0.97.3:37535): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
W20260923 21:05:19.441025 17104 leader_election.cc:341] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 800274ba5ea14fd29c64f3cf09fbcbf0 (127.0.97.2:45893): Not found: Tablet not found: a7f646e3e5b34ede8a22f6b493ce5de1
I20260923 21:05:19.441130 17104 leader_election.cc:302] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5625f3fe57844f31bd14aff52ddb1a4f; no voters: 5a13b2881b2048998fbbc0d63be1ac23, 800274ba5ea14fd29c64f3cf09fbcbf0
I20260923 21:05:19.441514 17535 raft_consensus.cc:2726] T a7f646e3e5b34ede8a22f6b493ce5de1 P 5625f3fe57844f31bd14aff52ddb1a4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 21:05:19.601024 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:19.600223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.2:50585)
0923 21:05:19.600784 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:19.600861 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":317,"threads_started":1}
I20260923 21:05:20.045205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 17048
I20260923 21:05:20.060552 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:20.059502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.1:58899)
0923 21:05:20.060174 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:20.060315 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":318,"threads_started":1}
I20260923 21:05:20.060910 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:20.059604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45325 (local address 127.0.97.3:40979)
0923 21:05:20.060457 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:20.060572 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":563,"threads_started":1}
I20260923 21:05:20.067186 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:20.046566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.97.1:43539)
0923 21:05:20.047238 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:20.047246 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:20.047275 (+ 29us) server_negotiation.cc:407] Connection header received
0923 21:05:20.047364 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:20.047379 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:20.047717 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:20.047907 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:20.049486 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:20.050554 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:20.051888 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:20.052483 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:20.052641 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:20.061170 (+ 8529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:20.061185 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:20.061203 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:20.061239 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:20.064464 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:20.065191 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:20.065199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:20.065201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:20.065351 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:20.065679 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:20.065682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:20.065684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:20.066070 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:20.066301 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:20.066686 (+ 385us) server_negotiation.cc:299] Negotiation successful
0923 21:05:20.066916 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":350,"threads_started":1}
I20260923 21:05:20.068804 17213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:20.163618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 17048
I20260923 21:05:20.182545 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 17222
I20260923 21:05:20.197499 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:20.183398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.0.97.2:45893)
0923 21:05:20.183920 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:20.183925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:20.183943 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:05:20.183999 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:20.184002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:20.184064 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:20.184163 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:20.185962 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:20.186884 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:20.188214 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:20.188652 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:20.188761 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:20.191731 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:20.191736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:20.191740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:20.191805 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:20.194953 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:20.195672 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:20.195675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:20.195678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:20.195863 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:20.196197 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:20.196199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:20.196201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:20.196481 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:20.196692 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:20.197020 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 21:05:20.197201 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":264,"threads_started":1}
I20260923 21:05:20.198303 17345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:20.289505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 17222
I20260923 21:05:20.308169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 17353
I20260923 21:05:20.323264 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:20.309100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.0.97.3:37535)
0923 21:05:20.309728 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:20.309732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:20.309751 (+ 19us) server_negotiation.cc:407] Connection header received
0923 21:05:20.309809 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:20.309812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:20.309880 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:20.309973 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:20.311616 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:20.312434 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:20.313848 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:20.314433 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:20.314544 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:20.317415 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:20.317421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:20.317425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:20.317464 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:20.320509 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:20.321323 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:20.321328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:20.321330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:20.321464 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:20.321801 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:20.321805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:20.321807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:20.322090 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:20.322548 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:20.322710 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 21:05:20.322955 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":366,"threads_started":1}
I20260923 21:05:20.324018 17476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:20.413614 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 17353
2026-09-23T21:05:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64917 ms)
[----------] 30 tests from SecurityITest (312418 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:20 dist-test-slave-8xn5 krb5kdc[17547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:20 dist-test-slave-8xn5 krb5kdc[17547](info): set up 2 sockets
Sep 23 21:05:20 dist-test-slave-8xn5 krb5kdc[17547](info): commencing operation
krb5kdc: starting...
W20260923 21:05:21.571298 388 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:21 dist-test-slave-8xn5 krb5kdc[17547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:05:21.851073 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44011
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:22.223592 17560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:22.224100 17560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:22.224355 17560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:22.235065 17560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:22.235209 17560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:22.235327 17560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:22.235447 17560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:22.235555 17560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:22.247269 17560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:22.248744 17560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:22.250134 17560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:22.263506 17565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:22.263453 17568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:22.263423 17566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:22.264276 17560 server_base.cc:1034] running on GCE node
I20260923 21:05:22.265125 17560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:22.266782 17560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:22.268007 17560 hybrid_clock.cc:648] HybridClock initialized: now 1790197522267963 us; error 40 us; skew 500 ppm
I20260923 21:05:22.268440 17560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:22 dist-test-slave-8xn5 krb5kdc[17547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:22.271951 17560 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:05:22.273736 17560 webserver.cc:460] Webserver started at http://127.0.97.62:44535/ using document root <none> and password file <none>
I20260923 21:05:22.274600 17560 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:22.274762 17560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:22.275103 17560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:22.278563 17560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "8b8a79dede9b4f69b1120ea13fdeae98"
format_stamp: "Formatted at 2026-09-23 21:05:22 on dist-test-slave-8xn5"
server_key: "396bade588ead868d97db0149d8807ea"
server_key_iv: "4cd52cfea7b5ded9fc3a45a8b5cd56f2"
server_key_version: "encryptionkey@0"
I20260923 21:05:22.279492 17560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b8a79dede9b4f69b1120ea13fdeae98"
format_stamp: "Formatted at 2026-09-23 21:05:22 on dist-test-slave-8xn5"
server_key: "396bade588ead868d97db0149d8807ea"
server_key_iv: "4cd52cfea7b5ded9fc3a45a8b5cd56f2"
server_key_version: "encryptionkey@0"
I20260923 21:05:22.285719 17560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 21:05:22.290104 17575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:22.291843 17560 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260923 21:05:22.292099 17560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "8b8a79dede9b4f69b1120ea13fdeae98"
format_stamp: "Formatted at 2026-09-23 21:05:22 on dist-test-slave-8xn5"
server_key: "396bade588ead868d97db0149d8807ea"
server_key_iv: "4cd52cfea7b5ded9fc3a45a8b5cd56f2"
server_key_version: "encryptionkey@0"
I20260923 21:05:22.292361 17560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:22.311800 17560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:22.316846 17560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:22.317186 17560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:22.337958 17560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44011
I20260923 21:05:22.338007 17626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44011 every 8 connection(s)
I20260923 21:05:22.339520 17560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:22.343698 17627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:22.348165 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 17560
I20260923 21:05:22.348417 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:22.349102 388 external_mini_cluster.cc:1428] Setting key 134187cfa2c0f242f3579a3eb7a22dc0
I20260923 21:05:22.357043 17627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98: Bootstrap starting.
I20260923 21:05:22.360948 17627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:22.362270 17627 log.cc:824] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:22.365540 17627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98: No bootstrap required, opened a new log
Sep 23 21:05:22 dist-test-slave-8xn5 krb5kdc[17547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:22.375092 17627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 21:05:22.375442 17630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:22.351289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.0.97.62:44011)
0923 21:05:22.352175 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:22.352193 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:22.352249 (+ 56us) server_negotiation.cc:407] Connection header received
0923 21:05:22.355211 (+ 2962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:22.355245 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:22.355817 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:22.356310 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:22.357788 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:22.359909 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:22.361465 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:22.362577 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:22.362779 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:22.367478 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:22.367486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:22.367504 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:22.367544 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:22.370948 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:22.371875 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:22.371881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:22.371888 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:22.372067 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:22.372480 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:22.372483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:22.372485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:22.373076 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:22.373328 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:22.373858 (+ 530us) server_negotiation.cc:299] Negotiation successful
0923 21:05:22.374316 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":362,"threads_started":1}
I20260923 21:05:22.375481 17627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:22.375607 17627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8a79dede9b4f69b1120ea13fdeae98, State: Initialized, Role: FOLLOWER
I20260923 21:05:22.376242 17627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 21:05:22.376515 17627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:22.376679 17627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:22.376868 17627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:22.379513 17627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 21:05:22.380131 17627 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8a79dede9b4f69b1120ea13fdeae98; no voters:
I20260923 21:05:22.381064 17627 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:22.381161 17632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:22.382310 17632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [term 1 LEADER]: Becoming Leader. State: Replica: 8b8a79dede9b4f69b1120ea13fdeae98, State: Running, Role: LEADER
I20260923 21:05:22.382982 17632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } }
I20260923 21:05:22.384124 17627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:22.386408 17634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } } }
I20260923 21:05:22.386579 17633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8a79dede9b4f69b1120ea13fdeae98. Latest consensus state: current_term: 1 leader_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8a79dede9b4f69b1120ea13fdeae98" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44011 } } }
I20260923 21:05:22.386890 17634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:22.386894 17633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:22.391726 17639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:22.397035 17639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:22.410574 17639 catalog_manager.cc:1347] Generated new cluster ID: e19934940c134f918e57f71d4f169603
I20260923 21:05:22.410748 17639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:22.422466 17639 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:22.424526 17639 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:22.432227 17639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98: Generated new TSK 0
I20260923 21:05:22.433223 17639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:05:22.511881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:22.879793 17653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:22.880087 17653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:22.880285 17653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:22.888191 17653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:22.888348 17653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:22.888581 17653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:05:22.899370 17653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:22.900704 17653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:22.901988 17653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:22.917384 17658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:22.917446 17659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:22.917325 17661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:22.918206 17653 server_base.cc:1034] running on GCE node
I20260923 21:05:22.918918 17653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:22.920015 17653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:22.921271 17653 hybrid_clock.cc:648] HybridClock initialized: now 1790197522921200 us; error 58 us; skew 500 ppm
I20260923 21:05:22.921612 17653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:22 dist-test-slave-8xn5 krb5kdc[17547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:22.925161 17653 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:05:22.926915 17653 webserver.cc:460] Webserver started at http://127.0.97.1:37121/ using document root <none> and password file <none>
I20260923 21:05:22.927685 17653 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:22.927843 17653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:22.928205 17653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:22.931718 17653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "35a3102b83c64b138cb32be08b81887c"
format_stamp: "Formatted at 2026-09-23 21:05:22 on dist-test-slave-8xn5"
server_key: "f8c547eef404b4101966b8d602dc7f06"
server_key_iv: "03e17fe7b54552e12ebdddaf0dbb1190"
server_key_version: "encryptionkey@0"
I20260923 21:05:22.932638 17653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "35a3102b83c64b138cb32be08b81887c"
format_stamp: "Formatted at 2026-09-23 21:05:22 on dist-test-slave-8xn5"
server_key: "f8c547eef404b4101966b8d602dc7f06"
server_key_iv: "03e17fe7b54552e12ebdddaf0dbb1190"
server_key_version: "encryptionkey@0"
I20260923 21:05:22.938599 17653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:05:22.943017 17668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:22.944756 17653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 21:05:22.944985 17653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "35a3102b83c64b138cb32be08b81887c"
format_stamp: "Formatted at 2026-09-23 21:05:22 on dist-test-slave-8xn5"
server_key: "f8c547eef404b4101966b8d602dc7f06"
server_key_iv: "03e17fe7b54552e12ebdddaf0dbb1190"
server_key_version: "encryptionkey@0"
I20260923 21:05:22.945254 17653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:22.979445 17653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:22.985183 17653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:22.985558 17653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:22.986781 17653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:22.988458 17653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:22.988586 17653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:22.988762 17653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:22.988869 17653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:23.028167 17653 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44239
I20260923 21:05:23.028280 17780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44239 every 8 connection(s)
I20260923 21:05:23.029601 17653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:05:23.033524 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 17653
I20260923 21:05:23.033761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:05:23.034641 388 external_mini_cluster.cc:1428] Setting key d2ef6dc4de2e9e3a334c92fc28f6552c
Sep 23 21:05:23 dist-test-slave-8xn5 krb5kdc[17547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:23.052453 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:23.032624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36021 (local address 127.0.97.62:44011)
0923 21:05:23.033183 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:23.033188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:23.034218 (+ 1030us) server_negotiation.cc:407] Connection header received
0923 21:05:23.036394 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:23.036399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:23.036541 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:23.036666 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:23.038288 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.039132 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.040877 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.041336 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.041506 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:23.045553 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:23.045558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:23.045562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:23.045597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:23.048573 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:23.049709 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:23.049715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:23.049719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:23.049919 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:23.050456 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:23.050462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:23.050465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:23.050779 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:23.050960 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:23.051798 (+ 838us) server_negotiation.cc:299] Negotiation successful
0923 21:05:23.052065 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":318,"threads_started":1}
I20260923 21:05:23.053591 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:23.032971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44011 (local address 127.0.97.1:36021)
0923 21:05:23.034028 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:23.034081 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:23.036109 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:23.036923 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:23.036933 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:23.037475 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:23.038095 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:23.038113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.039323 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.039333 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:23.040697 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:23.040711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.041634 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.041641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:23.041894 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:23.042897 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:23.042922 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:23.045274 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:23.048796 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:23.048805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:23.048821 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:23.049554 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:23.050048 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:23.050053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:23.050056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:23.050277 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:23.050960 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:23.050970 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:23.051380 (+ 410us) client_negotiation.cc:769] Sending connection context
0923 21:05:23.051675 (+ 295us) client_negotiation.cc:240] Negotiation successful
0923 21:05:23.052303 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":366,"threads_started":1}
I20260923 21:05:23.055472 17781 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44011
I20260923 21:05:23.055907 17781 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:23.056700 17781 heartbeater.cc:511] Master 127.0.97.62:44011 requested a full tablet report, sending...
I20260923 21:05:23.059402 17592 ts_manager.cc:194] Registered new tserver with Master: 35a3102b83c64b138cb32be08b81887c (127.0.97.1:44239)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:05:23.061218 17592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:05:23.099414 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:23.464284 17787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:23.464771 17787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:23.465013 17787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:23.475543 17787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:23.475728 17787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:23.475979 17787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:05:23.488454 17787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:23.490073 17787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:23.491834 17787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:23.507386 17795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:23.507407 17793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:23.507771 17792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:23.508325 17787 server_base.cc:1034] running on GCE node
I20260923 21:05:23.509035 17787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:23.510155 17787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:23.511451 17787 hybrid_clock.cc:648] HybridClock initialized: now 1790197523511398 us; error 59 us; skew 500 ppm
I20260923 21:05:23.511891 17787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:23 dist-test-slave-8xn5 krb5kdc[17547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:23.515436 17787 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:05:23.517341 17787 webserver.cc:460] Webserver started at http://127.0.97.2:43659/ using document root <none> and password file <none>
I20260923 21:05:23.518128 17787 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:23.518291 17787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:23.518653 17787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:23.521840 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0"
format_stamp: "Formatted at 2026-09-23 21:05:23 on dist-test-slave-8xn5"
server_key: "ad9bd2fbeea1af14b05de05e5314b551"
server_key_iv: "f166b1aec018948792103d5c92f7ddf3"
server_key_version: "encryptionkey@0"
I20260923 21:05:23.522794 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0"
format_stamp: "Formatted at 2026-09-23 21:05:23 on dist-test-slave-8xn5"
server_key: "ad9bd2fbeea1af14b05de05e5314b551"
server_key_iv: "f166b1aec018948792103d5c92f7ddf3"
server_key_version: "encryptionkey@0"
I20260923 21:05:23.528640 17787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260923 21:05:23.532932 17802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:23.534873 17787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260923 21:05:23.535094 17787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0"
format_stamp: "Formatted at 2026-09-23 21:05:23 on dist-test-slave-8xn5"
server_key: "ad9bd2fbeea1af14b05de05e5314b551"
server_key_iv: "f166b1aec018948792103d5c92f7ddf3"
server_key_version: "encryptionkey@0"
I20260923 21:05:23.535360 17787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:23.549106 17787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:23.554214 17787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:23.554595 17787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:23.555747 17787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:23.557294 17787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:23.557407 17787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:23.557574 17787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:23.557651 17787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:23.594784 17787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33927
I20260923 21:05:23.594874 17914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33927 every 8 connection(s)
I20260923 21:05:23.596194 17787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:05:23.597450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 17787
I20260923 21:05:23.597630 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:05:23.598246 388 external_mini_cluster.cc:1428] Setting key 87b1f8d1c48b853e9a77ca74793e9f7b
Sep 23 21:05:23 dist-test-slave-8xn5 krb5kdc[17547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:23.620242 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:23.599366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57105 (local address 127.0.97.62:44011)
0923 21:05:23.599960 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:23.599967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:23.601178 (+ 1211us) server_negotiation.cc:407] Connection header received
0923 21:05:23.603315 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:23.603321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:23.603419 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:23.603563 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:23.605273 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.606124 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.608459 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.609047 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.609157 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:23.614004 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:23.614010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:23.614014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:23.614051 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:23.616826 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:23.617698 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:23.617703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:23.617705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:23.617836 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:23.618316 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:23.618321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:23.618374 (+ 53us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:23.618679 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:23.618847 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:23.619602 (+ 755us) server_negotiation.cc:299] Negotiation successful
0923 21:05:23.619852 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":351,"threads_started":1}
I20260923 21:05:23.621276 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:23.600004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44011 (local address 127.0.97.2:57105)
0923 21:05:23.600962 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:23.601054 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:23.603023 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:23.603820 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:23.603829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:23.604429 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:23.605078 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:23.605098 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.606309 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.606322 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:23.608269 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:23.608296 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:23.609309 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:23.609319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:23.609689 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:23.610616 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:23.610654 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:23.613714 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:23.617011 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:23.617021 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:23.617040 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:23.617529 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:23.617962 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:23.617967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:23.617971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:23.618182 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:23.618819 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:23.618827 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:23.619210 (+ 383us) client_negotiation.cc:769] Sending connection context
0923 21:05:23.619496 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 21:05:23.620063 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":410,"threads_started":1}
I20260923 21:05:23.623148 17915 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44011
I20260923 21:05:23.623490 17915 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260923 21:05:23.624220 17915 heartbeater.cc:511] Master 127.0.97.62:44011 requested a full tablet report, sending...
I20260923 21:05:23.625833 17592 ts_manager.cc:194] Registered new tserver with Master: 16e5f26dd3a14f4485e14a3dfe0e0bf0 (127.0.97.2:33927)
I20260923 21:05:23.626747 17592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:05:23.662887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:24.011396 17921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:24.011736 17921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:24.011837 17921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:24.019387 17921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:24.019516 17921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:24.019631 17921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:05:24.029814 17921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:24.031214 17921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:24.032652 17921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:24.046567 17926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:24.046674 17927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:24.047097 17929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:24.047888 17921 server_base.cc:1034] running on GCE node
I20260923 21:05:24.048643 17921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:24.049803 17921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:24.051090 17921 hybrid_clock.cc:648] HybridClock initialized: now 1790197524050978 us; error 84 us; skew 500 ppm
I20260923 21:05:24.051646 17921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:24 dist-test-slave-8xn5 krb5kdc[17547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:24.055718 17921 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:05:24.057384 17921 webserver.cc:460] Webserver started at http://127.0.97.3:38187/ using document root <none> and password file <none>
I20260923 21:05:24.058102 17921 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:24.058213 17921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:24.058578 17921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:24.061736 17921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "d438984208e94b23b5ab0eff250cb327"
format_stamp: "Formatted at 2026-09-23 21:05:24 on dist-test-slave-8xn5"
server_key: "07b372f0754a65133452bdb9e2f00dad"
server_key_iv: "2c19ed0d2195eda2f11f4fcc0fe17070"
server_key_version: "encryptionkey@0"
I20260923 21:05:24.062634 17921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d438984208e94b23b5ab0eff250cb327"
format_stamp: "Formatted at 2026-09-23 21:05:24 on dist-test-slave-8xn5"
server_key: "07b372f0754a65133452bdb9e2f00dad"
server_key_iv: "2c19ed0d2195eda2f11f4fcc0fe17070"
server_key_version: "encryptionkey@0"
I20260923 21:05:24.064924 17781 heartbeater.cc:503] Master 127.0.97.62:44011 was elected leader, sending a full tablet report...
I20260923 21:05:24.068786 17921 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:05:24.073081 17936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:24.074999 17921 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:05:24.075199 17921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "d438984208e94b23b5ab0eff250cb327"
format_stamp: "Formatted at 2026-09-23 21:05:24 on dist-test-slave-8xn5"
server_key: "07b372f0754a65133452bdb9e2f00dad"
server_key_iv: "2c19ed0d2195eda2f11f4fcc0fe17070"
server_key_version: "encryptionkey@0"
I20260923 21:05:24.075449 17921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:24.101078 17921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:24.106141 17921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:24.106510 17921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:24.107631 17921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:24.109338 17921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:24.109421 17921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:24.109591 17921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:24.109656 17921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:24.148083 17921 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39835
I20260923 21:05:24.148178 18048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39835 every 8 connection(s)
I20260923 21:05:24.149657 17921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:05:24.159415 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 17921
I20260923 21:05:24.159607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:05:24.160135 388 external_mini_cluster.cc:1428] Setting key 2d9958da5f604f391e789793c8da2787
Sep 23 21:05:24 dist-test-slave-8xn5 krb5kdc[17547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:24.172654 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.152805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59747 (local address 127.0.97.62:44011)
0923 21:05:24.153373 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.153377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.154692 (+ 1315us) server_negotiation.cc:407] Connection header received
0923 21:05:24.156583 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.156590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.156684 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.156819 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:24.158394 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.159186 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.161120 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.161759 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.161888 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.165982 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:24.165991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:24.165994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:24.166032 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:24.169277 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.170046 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.170051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.170054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.170200 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.170799 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.170803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.170805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.171098 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:24.171227 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.172128 (+ 901us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.172323 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":249,"threads_started":1}
I20260923 21:05:24.173718 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.153583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44011 (local address 127.0.97.3:59747)
0923 21:05:24.154493 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:24.154535 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:24.156293 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:24.157067 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:24.157076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:24.157617 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:24.158197 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.158216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.159348 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.159354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.160968 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.160981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.161996 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.162003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.162294 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.163118 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:24.163147 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:24.165735 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:24.169448 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:24.169455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:24.169469 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:24.169917 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:24.170313 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:24.170316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:24.170318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:24.170640 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:24.171255 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:24.171262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:24.171736 (+ 474us) client_negotiation.cc:769] Sending connection context
0923 21:05:24.172009 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 21:05:24.172442 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":303,"threads_started":1}
I20260923 21:05:24.175877 18049 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44011
I20260923 21:05:24.176288 18049 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:24.177194 18049 heartbeater.cc:511] Master 127.0.97.62:44011 requested a full tablet report, sending...
I20260923 21:05:24.178695 17592 ts_manager.cc:194] Registered new tserver with Master: d438984208e94b23b5ab0eff250cb327 (127.0.97.3:39835)
I20260923 21:05:24.179705 17592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59747
I20260923 21:05:24.190194 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:05:24.210323 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.195735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.0.97.62:44011)
0923 21:05:24.195962 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.195992 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.196434 (+ 442us) server_negotiation.cc:407] Connection header received
0923 21:05:24.196605 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.196611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.196697 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.196804 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:24.198454 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.199268 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.200622 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.201060 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.201167 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.203792 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:24.203797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:24.203800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:24.203835 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:24.206950 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.207718 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.207723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.207725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.207918 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.208351 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.208355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.208357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.208845 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:24.209103 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.209563 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.209843 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 21:05:24.218438 17592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:05:24.225468 17592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:05:24.252508 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.239374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33927 (local address 127.0.0.1:46908)
0923 21:05:24.241450 (+ 2076us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:24.241501 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:24.241692 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:24.244076 (+ 2384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:24.244083 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:24.244123 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:24.244603 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.244615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.246927 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.246932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.249713 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.249729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.251367 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.251374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.251595 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.251672 (+ 77us) client_negotiation.cc:769] Sending connection context
0923 21:05:24.251812 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 21:05:24.252016 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1837,"mutex_wait_us":12,"thread_start_us":1647,"threads_started":1}
I20260923 21:05:24.253494 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.240172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.0.97.3:39835)
0923 21:05:24.243003 (+ 2831us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.243016 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.243448 (+ 432us) server_negotiation.cc:407] Connection header received
0923 21:05:24.243564 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.243572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.243832 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.244054 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:24.245062 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.246780 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.250316 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.251866 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.251999 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.252253 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.252935 (+ 682us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.253122 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2591,"thread_start_us":2405,"threads_started":1}
I20260923 21:05:24.253515 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.239812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.0.97.2:33927)
0923 21:05:24.243437 (+ 3625us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.243445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.243469 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:05:24.243564 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.243572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.243832 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.244822 (+ 990us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:24.245062 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.246763 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.249861 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.251162 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.252612 (+ 1450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.252939 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.253015 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.253235 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3484,"thread_start_us":3367,"threads_started":1}
I20260923 21:05:24.254304 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.239664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39835 (local address 127.0.0.1:34848)
0923 21:05:24.243262 (+ 3598us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:24.243280 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:24.243397 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:24.244353 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:24.244357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:24.244377 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:24.244838 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.244850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.246927 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.246932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.250145 (+ 3213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.250161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.252407 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.252415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.252615 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.252650 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:05:24.253763 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0923 21:05:24.253942 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3481,"mutex_wait_us":572,"thread_start_us":3344,"threads_started":1}
I20260923 21:05:24.257926 18066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.243736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44239 (local address 127.0.0.1:50202)
0923 21:05:24.248053 (+ 4317us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:24.248077 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:24.248223 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:24.248939 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:24.248945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:24.248970 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:24.249444 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.249457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.251354 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.251360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.253592 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.253609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.257123 (+ 3514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.257130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.257309 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.257343 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 21:05:24.257437 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 21:05:24.257589 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4168,"thread_start_us":4016,"threads_started":1}
I20260923 21:05:24.258177 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.244107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.0.97.1:44239)
0923 21:05:24.247545 (+ 3438us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.247554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.248370 (+ 816us) server_negotiation.cc:407] Connection header received
0923 21:05:24.248486 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.248495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.248766 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.248955 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:24.249620 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.251180 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.254759 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.256918 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.257091 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.257331 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.257518 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.257704 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3195,"thread_start_us":3018,"threads_started":1}
I20260923 21:05:24.257917 17850 tablet_service.cc:1467] Processing CreateTablet for tablet faa2fc73cbf1410a8d5910e02842bcd9 (DEFAULT_TABLE table=test-table [id=85f682a18d1445009fd0fa7e43783970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:24.258272 17984 tablet_service.cc:1467] Processing CreateTablet for tablet faa2fc73cbf1410a8d5910e02842bcd9 (DEFAULT_TABLE table=test-table [id=85f682a18d1445009fd0fa7e43783970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:24.259600 17850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa2fc73cbf1410a8d5910e02842bcd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:24.259716 17984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa2fc73cbf1410a8d5910e02842bcd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:24.261821 17716 tablet_service.cc:1467] Processing CreateTablet for tablet faa2fc73cbf1410a8d5910e02842bcd9 (DEFAULT_TABLE table=test-table [id=85f682a18d1445009fd0fa7e43783970]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:24.263067 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa2fc73cbf1410a8d5910e02842bcd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:24.274572 18068 tablet_bootstrap.cc:492] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Bootstrap starting.
I20260923 21:05:24.277969 18068 tablet_bootstrap.cc:654] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:24.278142 18070 tablet_bootstrap.cc:492] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Bootstrap starting.
I20260923 21:05:24.279214 18068 log.cc:824] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:24.283332 18069 tablet_bootstrap.cc:492] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Bootstrap starting.
I20260923 21:05:24.284526 18070 tablet_bootstrap.cc:654] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:24.285660 18070 log.cc:824] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:24.287357 18068 tablet_bootstrap.cc:492] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: No bootstrap required, opened a new log
I20260923 21:05:24.287669 18069 tablet_bootstrap.cc:654] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:24.287695 18068 ts_tablet_manager.cc:1397] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260923 21:05:24.288525 18070 tablet_bootstrap.cc:492] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: No bootstrap required, opened a new log
I20260923 21:05:24.288802 18070 ts_tablet_manager.cc:1397] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 21:05:24.289664 18069 log.cc:824] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:24.293144 18069 tablet_bootstrap.cc:492] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: No bootstrap required, opened a new log
I20260923 21:05:24.293490 18069 ts_tablet_manager.cc:1397] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260923 21:05:24.297528 18070 raft_consensus.cc:348] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.298026 18070 raft_consensus.cc:374] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:24.298195 18070 raft_consensus.cc:729] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d438984208e94b23b5ab0eff250cb327, State: Initialized, Role: FOLLOWER
I20260923 21:05:24.298919 18070 consensus_queue.cc:260] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.298951 18068 raft_consensus.cc:348] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.299425 18068 raft_consensus.cc:374] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:24.299535 18068 raft_consensus.cc:729] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a3102b83c64b138cb32be08b81887c, State: Initialized, Role: FOLLOWER
I20260923 21:05:24.300335 18068 consensus_queue.cc:260] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.305698 18068 ts_tablet_manager.cc:1428] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Time spent starting tablet: real 0.018s user 0.016s sys 0.001s
I20260923 21:05:24.305589 18069 raft_consensus.cc:348] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.306106 18069 raft_consensus.cc:374] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:24.306222 18069 raft_consensus.cc:729] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e5f26dd3a14f4485e14a3dfe0e0bf0, State: Initialized, Role: FOLLOWER
I20260923 21:05:24.306945 18069 consensus_queue.cc:260] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.308337 17915 heartbeater.cc:503] Master 127.0.97.62:44011 was elected leader, sending a full tablet report...
I20260923 21:05:24.309082 18069 ts_tablet_manager.cc:1428] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Time spent starting tablet: real 0.015s user 0.014s sys 0.003s
I20260923 21:05:24.310065 18049 heartbeater.cc:503] Master 127.0.97.62:44011 was elected leader, sending a full tablet report...
I20260923 21:05:24.310798 18070 ts_tablet_manager.cc:1428] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Time spent starting tablet: real 0.022s user 0.019s sys 0.005s
W20260923 21:05:24.348604 17916 tablet.cc:2369] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:24.401639 18050 tablet.cc:2369] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:24.485737 18074 raft_consensus.cc:482] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:24.486044 18074 raft_consensus.cc:504] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.487963 18074 leader_election.cc:288] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16e5f26dd3a14f4485e14a3dfe0e0bf0 (127.0.97.2:33927), 35a3102b83c64b138cb32be08b81887c (127.0.97.1:44239)
I20260923 21:05:24.495558 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.488690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35809 (local address 127.0.97.2:33927)
0923 21:05:24.488965 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.488974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.489331 (+ 357us) server_negotiation.cc:407] Connection header received
0923 21:05:24.489425 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.489431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.489522 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.489649 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:24.490232 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.491277 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.493463 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.494566 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.494682 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.494754 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.495128 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.495285 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 21:05:24.495651 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.488272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33927 (local address 127.0.97.3:35809)
0923 21:05:24.489007 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:24.489034 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:24.489176 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:24.489706 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:24.489712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:24.489775 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:24.490053 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.490068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.491417 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.491423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.493297 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.493312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.494731 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.494738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.494935 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.494986 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 21:05:24.495096 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 21:05:24.495267 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":87}
I20260923 21:05:24.496119 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.488777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45545 (local address 127.0.97.1:44239)
0923 21:05:24.489174 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.489180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.489699 (+ 519us) server_negotiation.cc:407] Connection header received
0923 21:05:24.489788 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.489793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.489876 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.489996 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:24.490688 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.491781 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.493994 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.495154 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.495276 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.495369 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.495765 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.495942 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260923 21:05:24.496852 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.488473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44239 (local address 127.0.97.3:45545)
0923 21:05:24.489449 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:24.489465 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:24.489581 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:24.490014 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:24.490017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:24.490037 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:24.490481 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.490495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.491953 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.491956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.493815 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:24.493829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.495368 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.495374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:24.495546 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.495587 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 21:05:24.496329 (+ 742us) client_negotiation.cc:240] Negotiation successful
0923 21:05:24.496474 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":25,"thread_start_us":646,"threads_started":1}
I20260923 21:05:24.496833 17870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa2fc73cbf1410a8d5910e02842bcd9" candidate_uuid: "d438984208e94b23b5ab0eff250cb327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" is_pre_election: true
I20260923 21:05:24.497450 17870 raft_consensus.cc:2454] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d438984208e94b23b5ab0eff250cb327 in term 0.
I20260923 21:05:24.497982 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa2fc73cbf1410a8d5910e02842bcd9" candidate_uuid: "d438984208e94b23b5ab0eff250cb327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a3102b83c64b138cb32be08b81887c" is_pre_election: true
I20260923 21:05:24.498376 17937 leader_election.cc:302] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e5f26dd3a14f4485e14a3dfe0e0bf0, d438984208e94b23b5ab0eff250cb327; no voters:
I20260923 21:05:24.498505 17736 raft_consensus.cc:2454] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d438984208e94b23b5ab0eff250cb327 in term 0.
I20260923 21:05:24.498834 18074 raft_consensus.cc:2781] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:05:24.499042 18074 raft_consensus.cc:482] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:05:24.499166 18074 raft_consensus.cc:3037] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:24.501434 18074 raft_consensus.cc:504] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.502104 18074 leader_election.cc:288] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [CANDIDATE]: Term 1 election: Requested vote from peers 16e5f26dd3a14f4485e14a3dfe0e0bf0 (127.0.97.2:33927), 35a3102b83c64b138cb32be08b81887c (127.0.97.1:44239)
I20260923 21:05:24.502655 17736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa2fc73cbf1410a8d5910e02842bcd9" candidate_uuid: "d438984208e94b23b5ab0eff250cb327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a3102b83c64b138cb32be08b81887c"
I20260923 21:05:24.502804 17870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa2fc73cbf1410a8d5910e02842bcd9" candidate_uuid: "d438984208e94b23b5ab0eff250cb327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0"
I20260923 21:05:24.502893 17736 raft_consensus.cc:3037] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:24.503026 17870 raft_consensus.cc:3037] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:24.506258 17736 raft_consensus.cc:2454] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d438984208e94b23b5ab0eff250cb327 in term 1.
I20260923 21:05:24.506258 17870 raft_consensus.cc:2454] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d438984208e94b23b5ab0eff250cb327 in term 1.
I20260923 21:05:24.506747 17937 leader_election.cc:302] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e5f26dd3a14f4485e14a3dfe0e0bf0, d438984208e94b23b5ab0eff250cb327; no voters:
I20260923 21:05:24.507103 18074 raft_consensus.cc:2781] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:24.507726 18074 raft_consensus.cc:686] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 1 LEADER]: Becoming Leader. State: Replica: d438984208e94b23b5ab0eff250cb327, State: Running, Role: LEADER
I20260923 21:05:24.508311 18074 consensus_queue.cc:237] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } }
I20260923 21:05:24.513273 17591 catalog_manager.cc:5495] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 reported cstate change: term changed from 0 to 1, leader changed from <none> to d438984208e94b23b5ab0eff250cb327 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d438984208e94b23b5ab0eff250cb327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d438984208e94b23b5ab0eff250cb327" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 } health_report { overall_health: UNKNOWN } } }
W20260923 21:05:24.532668 17782 tablet.cc:2369] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:24.560631 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.554482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.0.97.3:39835)
0923 21:05:24.554769 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.554776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.554802 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:05:24.555056 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.555061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.555169 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.555341 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:05:24.555761 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.556663 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.558379 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.558788 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.558940 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.559483 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:05:24.560027 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:05:24.560212 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.560272 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.560356 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260923 21:05:24.569314 17736 raft_consensus.cc:1261] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 1 FOLLOWER]: Refusing update from remote peer d438984208e94b23b5ab0eff250cb327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:24.569314 17870 raft_consensus.cc:1261] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 1 FOLLOWER]: Refusing update from remote peer d438984208e94b23b5ab0eff250cb327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:24.570374 18078 consensus_queue.cc:1035] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a3102b83c64b138cb32be08b81887c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:24.570822 18074 consensus_queue.cc:1035] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e5f26dd3a14f4485e14a3dfe0e0bf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:24.599043 17592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:24.599397 17592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:24.604197 17592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b8a79dede9b4f69b1120ea13fdeae98: Sending DeleteTablet for 3 replicas of tablet faa2fc73cbf1410a8d5910e02842bcd9
I20260923 21:05:24.605371 17984 tablet_service.cc:1514] Processing DeleteTablet for tablet faa2fc73cbf1410a8d5910e02842bcd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34848
I20260923 21:05:24.605422 17850 tablet_service.cc:1514] Processing DeleteTablet for tablet faa2fc73cbf1410a8d5910e02842bcd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46908
I20260923 21:05:24.610185 17716 tablet_service.cc:1514] Processing DeleteTablet for tablet faa2fc73cbf1410a8d5910e02842bcd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50202
I20260923 21:05:24.615526 18093 tablet_replica.cc:330] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: stopping tablet replica
I20260923 21:05:24.617189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 17653
I20260923 21:05:24.620868 18091 tablet_replica.cc:330] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: stopping tablet replica
I20260923 21:05:24.621392 18091 raft_consensus.cc:2229] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:05:24.622046 18093 raft_consensus.cc:2229] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:24.622206 18091 raft_consensus.cc:2258] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:24.623190 18092 tablet_replica.cc:330] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: stopping tablet replica
I20260923 21:05:24.623310 18093 raft_consensus.cc:2258] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:24.623617 18092 raft_consensus.cc:2229] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:24.624087 18092 raft_consensus.cc:2258] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:24.625015 18091 ts_tablet_manager.cc:1905] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:24.626044 18092 ts_tablet_manager.cc:1905] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:24.629632 18093 ts_tablet_manager.cc:1905] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:24.632623 18091 ts_tablet_manager.cc:1918] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:24.632807 18091 log.cc:1196] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/faa2fc73cbf1410a8d5910e02842bcd9
I20260923 21:05:24.633153 18092 ts_tablet_manager.cc:1918] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:24.633339 18092 log.cc:1196] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/faa2fc73cbf1410a8d5910e02842bcd9
I20260923 21:05:24.633409 18091 ts_tablet_manager.cc:1939] T faa2fc73cbf1410a8d5910e02842bcd9 P d438984208e94b23b5ab0eff250cb327: Deleting consensus metadata
I20260923 21:05:24.633903 18092 ts_tablet_manager.cc:1939] T faa2fc73cbf1410a8d5910e02842bcd9 P 16e5f26dd3a14f4485e14a3dfe0e0bf0: Deleting consensus metadata
I20260923 21:05:24.635421 17579 catalog_manager.cc:4841] TS d438984208e94b23b5ab0eff250cb327 (127.0.97.3:39835): tablet faa2fc73cbf1410a8d5910e02842bcd9 (table test-table [id=85f682a18d1445009fd0fa7e43783970]) successfully deleted
I20260923 21:05:24.635711 17576 catalog_manager.cc:4841] TS 16e5f26dd3a14f4485e14a3dfe0e0bf0 (127.0.97.2:33927): tablet faa2fc73cbf1410a8d5910e02842bcd9 (table test-table [id=85f682a18d1445009fd0fa7e43783970]) successfully deleted
Sep 23 21:05:24 dist-test-slave-8xn5 krb5kdc[17547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:05:24.639914 18093 ts_tablet_manager.cc:1918] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:24.640094 18093 log.cc:1196] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/faa2fc73cbf1410a8d5910e02842bcd9
I20260923 21:05:24.640575 18093 ts_tablet_manager.cc:1939] T faa2fc73cbf1410a8d5910e02842bcd9 P 35a3102b83c64b138cb32be08b81887c: Deleting consensus metadata
I20260923 21:05:24.642486 17579 catalog_manager.cc:4841] TS 35a3102b83c64b138cb32be08b81887c (127.0.97.1:44239): tablet faa2fc73cbf1410a8d5910e02842bcd9 (table test-table [id=85f682a18d1445009fd0fa7e43783970]) successfully deleted
I20260923 21:05:24.647387 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.618705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.0.97.1:44239)
0923 21:05:24.618872 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.618877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.621067 (+ 2190us) server_negotiation.cc:407] Connection header received
0923 21:05:24.621374 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.621381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.621492 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.621620 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:24.626438 (+ 4818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.627522 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.632881 (+ 5359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.633614 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.633745 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.640321 (+ 6576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:24.640338 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:24.640358 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:24.640411 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:24.644474 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.645188 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.645198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.645201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.645359 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.645722 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.645728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.645731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.646173 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:24.646525 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.646778 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.647004 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 21:05:24.648157 17776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:24.739871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 17653
I20260923 21:05:24.763810 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 17787
Sep 23 21:05:24 dist-test-slave-8xn5 krb5kdc[17547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:05:24.780748 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.764742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.0.97.2:33927)
0923 21:05:24.764920 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.764928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.764949 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:05:24.765211 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.765216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.765298 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.765417 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:24.766888 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.767661 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.769025 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.769423 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.769571 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.774584 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:24.774602 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:24.774620 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:24.774667 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:24.777641 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.778492 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.778500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.778502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.778636 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.779042 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.779052 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.779054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.779459 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:24.779800 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.780142 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.780370 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:05:24.781522 17910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:24.873435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 17787
I20260923 21:05:24.892045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 17921
Sep 23 21:05:24 dist-test-slave-8xn5 krb5kdc[17547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:05:24.910665 18065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:24.892886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.0.97.3:39835)
0923 21:05:24.893058 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:24.893067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:24.893174 (+ 107us) server_negotiation.cc:407] Connection header received
0923 21:05:24.893410 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:24.893416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:24.893503 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:24.893629 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:24.895025 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.895836 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.897275 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:24.897825 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:24.897973 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:24.903150 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:24.903163 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:24.903179 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:24.903223 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:24.906817 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.908085 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.908101 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.908104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.908274 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:24.908754 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:24.908760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:24.908763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:24.909171 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:24.909626 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:24.909931 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 21:05:24.910214 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 21:05:24.911495 18044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:25.036742 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 17921
I20260923 21:05:25.054189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 17560
I20260923 21:05:25.055732 17622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:25.169085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 17560
2026-09-23T21:05:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4764 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T21:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:25Z Disabled control of system clock
I20260923 21:05:25.255373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46443
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 21:05:25.617722 18102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:25.618031 18102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:25.618115 18102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:25.625993 18102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:25.626120 18102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:25.626158 18102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:25.626209 18102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:25.626253 18102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:25.636282 18102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:25.637629 18102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:25.640154 18102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:25.653024 18107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:25.652978 18108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:25.652933 18110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:25.654119 18102 server_base.cc:1034] running on GCE node
I20260923 21:05:25.655052 18102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:25.656736 18102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:25.657989 18102 hybrid_clock.cc:648] HybridClock initialized: now 1790197525657931 us; error 58 us; skew 500 ppm
I20260923 21:05:25.658445 18102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:25.667047 18102 webserver.cc:460] Webserver started at http://127.0.97.62:39613/ using document root <none> and password file <none>
I20260923 21:05:25.667801 18102 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:25.667899 18102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:25.668183 18102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:25.671363 18102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "173feef1b1fd4f398d9a85c79225682d"
format_stamp: "Formatted at 2026-09-23 21:05:25 on dist-test-slave-8xn5"
server_key: "2653596edfa2db7beba31a244000cf55"
server_key_iv: "7c7b4f024ba76438bf7ff40bbd251fa5"
server_key_version: "encryptionkey@0"
I20260923 21:05:25.672173 18102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "173feef1b1fd4f398d9a85c79225682d"
format_stamp: "Formatted at 2026-09-23 21:05:25 on dist-test-slave-8xn5"
server_key: "2653596edfa2db7beba31a244000cf55"
server_key_iv: "7c7b4f024ba76438bf7ff40bbd251fa5"
server_key_version: "encryptionkey@0"
I20260923 21:05:25.677824 18102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 21:05:25.681649 18116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:25.683248 18102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:05:25.683401 18102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "173feef1b1fd4f398d9a85c79225682d"
format_stamp: "Formatted at 2026-09-23 21:05:25 on dist-test-slave-8xn5"
server_key: "2653596edfa2db7beba31a244000cf55"
server_key_iv: "7c7b4f024ba76438bf7ff40bbd251fa5"
server_key_version: "encryptionkey@0"
I20260923 21:05:25.683590 18102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:25.703233 18102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:25.704071 18102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:25.704336 18102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:25.724802 18102 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46443
I20260923 21:05:25.724833 18167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46443 every 8 connection(s)
I20260923 21:05:25.726300 18102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:25.730322 18168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:25.732416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 18102
I20260923 21:05:25.732712 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:25.733196 388 external_mini_cluster.cc:1428] Setting key 0c797344f588f151c189300e6a2ae57f
I20260923 21:05:25.745003 18168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d: Bootstrap starting.
I20260923 21:05:25.749387 18168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:25.749987 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:25.735361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.0.97.62:46443)
0923 21:05:25.736562 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:25.736582 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:25.736641 (+ 59us) server_negotiation.cc:407] Connection header received
0923 21:05:25.739140 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:25.739175 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:25.739819 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:25.740293 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:25.740895 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:25.743025 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:25.744426 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:25.745484 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:25.745690 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:25.746931 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:25.746944 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:25.746966 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:25.747752 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:25.747871 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:25.747897 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:25.748025 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:25.748223 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 21:05:25.748596 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":351,"threads_started":1}
I20260923 21:05:25.750733 18168 log.cc:824] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:25.753700 18168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d: No bootstrap required, opened a new log
I20260923 21:05:25.761842 18168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173feef1b1fd4f398d9a85c79225682d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46443 } }
I20260923 21:05:25.762308 18168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:25.762434 18168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 173feef1b1fd4f398d9a85c79225682d, State: Initialized, Role: FOLLOWER
I20260923 21:05:25.763127 18168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173feef1b1fd4f398d9a85c79225682d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46443 } }
I20260923 21:05:25.763412 18168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:25.763569 18168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:25.763746 18168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:25.766033 18168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173feef1b1fd4f398d9a85c79225682d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46443 } }
I20260923 21:05:25.766681 18168 leader_election.cc:302] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 173feef1b1fd4f398d9a85c79225682d; no voters:
I20260923 21:05:25.767331 18168 leader_election.cc:288] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:25.767496 18173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:25.768365 18173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [term 1 LEADER]: Becoming Leader. State: Replica: 173feef1b1fd4f398d9a85c79225682d, State: Running, Role: LEADER
I20260923 21:05:25.768968 18173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173feef1b1fd4f398d9a85c79225682d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46443 } }
I20260923 21:05:25.769757 18168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:25.772310 18175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 173feef1b1fd4f398d9a85c79225682d. Latest consensus state: current_term: 1 leader_uuid: "173feef1b1fd4f398d9a85c79225682d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173feef1b1fd4f398d9a85c79225682d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46443 } } }
I20260923 21:05:25.772828 18175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:25.772306 18174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "173feef1b1fd4f398d9a85c79225682d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "173feef1b1fd4f398d9a85c79225682d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46443 } } }
I20260923 21:05:25.773031 18174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:25.775746 18178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:25.781831 18178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:25.792083 18178 catalog_manager.cc:1347] Generated new cluster ID: f04607b7f64f4c7a881eb44300f0b850
I20260923 21:05:25.792256 18178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:25.819236 18178 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:25.820716 18178 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:25.830797 18178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d: Generated new TSK 0
I20260923 21:05:25.831766 18178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:25.844239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46443
--builtin_ntp_servers=127.0.97.20:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 21:05:26.197185 18192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:26.197603 18192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:26.197777 18192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:26.207015 18192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:26.207211 18192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:26.207415 18192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:05:26.222832 18192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:26.224308 18192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:26.226470 18192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:26.241608 18198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:26.241704 18197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:26.242166 18200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:26.242659 18192 server_base.cc:1034] running on GCE node
I20260923 21:05:26.243357 18192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:26.244449 18192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:26.245645 18192 hybrid_clock.cc:648] HybridClock initialized: now 1790197526245570 us; error 68 us; skew 500 ppm
I20260923 21:05:26.245954 18192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:26.248688 18192 webserver.cc:460] Webserver started at http://127.0.97.1:46875/ using document root <none> and password file <none>
I20260923 21:05:26.249476 18192 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:26.249607 18192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:26.249967 18192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:26.253053 18192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "40aea24bc9fc4ac995cf76167aa0fca6"
format_stamp: "Formatted at 2026-09-23 21:05:26 on dist-test-slave-8xn5"
server_key: "6cc9b088cb272363769784d93fcea110"
server_key_iv: "170aff3c290f351861051d93bcb17dce"
server_key_version: "encryptionkey@0"
I20260923 21:05:26.253931 18192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "40aea24bc9fc4ac995cf76167aa0fca6"
format_stamp: "Formatted at 2026-09-23 21:05:26 on dist-test-slave-8xn5"
server_key: "6cc9b088cb272363769784d93fcea110"
server_key_iv: "170aff3c290f351861051d93bcb17dce"
server_key_version: "encryptionkey@0"
I20260923 21:05:26.260147 18192 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 21:05:26.264065 18206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:26.265540 18192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 21:05:26.265704 18192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "40aea24bc9fc4ac995cf76167aa0fca6"
format_stamp: "Formatted at 2026-09-23 21:05:26 on dist-test-slave-8xn5"
server_key: "6cc9b088cb272363769784d93fcea110"
server_key_iv: "170aff3c290f351861051d93bcb17dce"
server_key_version: "encryptionkey@0"
I20260923 21:05:26.266004 18192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:26.292814 18192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:26.293799 18192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:26.294127 18192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:26.295300 18192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:26.296941 18192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:26.297037 18192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:26.297178 18192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:26.297255 18192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:26.333734 18192 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35781
I20260923 21:05:26.333788 18318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35781 every 8 connection(s)
I20260923 21:05:26.335212 18192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:05:26.344390 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 18192
I20260923 21:05:26.344601 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:05:26.345095 388 external_mini_cluster.cc:1428] Setting key 46e39aa2e10d09495cbdaef315e48b3a
I20260923 21:05:26.348311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46443
--builtin_ntp_servers=127.0.97.20:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 21:05:26.350370 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:26.338398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54395 (local address 127.0.97.62:46443)
0923 21:05:26.339046 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:26.339050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:26.340144 (+ 1094us) server_negotiation.cc:407] Connection header received
0923 21:05:26.342361 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:26.342368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:26.342467 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:26.342606 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:26.343822 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.344763 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.346909 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.347503 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.347676 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:26.349284 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:26.349294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:26.349302 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:26.349329 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:26.349382 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:26.349402 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:26.349504 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:26.349770 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 21:05:26.349889 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":257,"threads_started":1}
I20260923 21:05:26.351414 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:26.338915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46443 (local address 127.0.97.1:54395)
0923 21:05:26.339944 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:26.339996 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:26.341984 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:26.342880 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:26.342889 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:26.343002 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:26.343628 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:26.343645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.344926 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.344938 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:26.346715 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:26.346731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.347808 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.347814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:26.348052 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:26.348911 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:26.348914 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:26.348959 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:26.349058 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:26.349538 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:26.349547 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:26.349554 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 21:05:26.349721 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 21:05:26.350079 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":283,"threads_started":1}
I20260923 21:05:26.353245 18319 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46443
I20260923 21:05:26.353602 18319 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:26.354526 18319 heartbeater.cc:511] Master 127.0.97.62:46443 requested a full tablet report, sending...
I20260923 21:05:26.356781 18133 ts_manager.cc:194] Registered new tserver with Master: 40aea24bc9fc4ac995cf76167aa0fca6 (127.0.97.1:35781)
I20260923 21:05:26.358569 18133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54395
W20260923 21:05:26.689810 18323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:26.690124 18323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:26.690268 18323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:26.697782 18323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:26.697898 18323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:26.698083 18323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:05:26.707653 18323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:26.709019 18323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:26.710886 18323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:26.725004 18329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:26.725024 18331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:26.725587 18328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:26.726373 18323 server_base.cc:1034] running on GCE node
I20260923 21:05:26.727087 18323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:26.728190 18323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:26.729425 18323 hybrid_clock.cc:648] HybridClock initialized: now 1790197526729377 us; error 44 us; skew 500 ppm
I20260923 21:05:26.729771 18323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:26.732352 18323 webserver.cc:460] Webserver started at http://127.0.97.2:44029/ using document root <none> and password file <none>
I20260923 21:05:26.733209 18323 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:26.733333 18323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:26.733673 18323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:26.736696 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "e3fe6a31d4b74c98a3f758fb0a74005c"
format_stamp: "Formatted at 2026-09-23 21:05:26 on dist-test-slave-8xn5"
server_key: "fbcae9478e8246e27f7ad73e22f5ede4"
server_key_iv: "3304ee2cefd39871a0e064531e4e2bfa"
server_key_version: "encryptionkey@0"
I20260923 21:05:26.737532 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e3fe6a31d4b74c98a3f758fb0a74005c"
format_stamp: "Formatted at 2026-09-23 21:05:26 on dist-test-slave-8xn5"
server_key: "fbcae9478e8246e27f7ad73e22f5ede4"
server_key_iv: "3304ee2cefd39871a0e064531e4e2bfa"
server_key_version: "encryptionkey@0"
I20260923 21:05:26.743264 18323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260923 21:05:26.746932 18337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:26.748744 18323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:05:26.748948 18323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "e3fe6a31d4b74c98a3f758fb0a74005c"
format_stamp: "Formatted at 2026-09-23 21:05:26 on dist-test-slave-8xn5"
server_key: "fbcae9478e8246e27f7ad73e22f5ede4"
server_key_iv: "3304ee2cefd39871a0e064531e4e2bfa"
server_key_version: "encryptionkey@0"
I20260923 21:05:26.749167 18323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:26.763128 18323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:26.764182 18323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:26.764575 18323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:26.765934 18323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:26.767916 18323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:26.768018 18323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:26.768177 18323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:26.768249 18323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:26.809716 18323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38997
I20260923 21:05:26.809772 18449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38997 every 8 connection(s)
I20260923 21:05:26.811188 18323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:05:26.814849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 18323
I20260923 21:05:26.815034 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:05:26.815514 388 external_mini_cluster.cc:1428] Setting key d1e0c36da4a86cc85550fd1408dfc7ce
I20260923 21:05:26.818392 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46443
--builtin_ntp_servers=127.0.97.20:41545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 21:05:26.825083 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:26.814071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47679 (local address 127.0.97.62:46443)
0923 21:05:26.814304 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:26.814309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:26.816123 (+ 1814us) server_negotiation.cc:407] Connection header received
0923 21:05:26.817919 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:26.817924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:26.818024 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:26.818159 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:26.819462 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.820218 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.821855 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.822283 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.822509 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:26.824152 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:26.824162 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:26.824166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:26.824188 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:26.824223 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:26.824237 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:26.824350 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:26.824605 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 21:05:26.824716 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 21:05:26.826318 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:26.814721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46443 (local address 127.0.97.2:47679)
0923 21:05:26.815927 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:26.815971 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:26.817637 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:26.818483 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:26.818492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:26.818646 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:26.819278 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:26.819297 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.820388 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.820393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:26.821696 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:26.821708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:26.822613 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:26.822620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:26.822860 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:26.823699 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:26.823701 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:26.823744 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:26.823831 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:26.824350 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:26.824364 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:26.824374 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 21:05:26.824597 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 21:05:26.825043 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":432,"threads_started":1}
I20260923 21:05:26.828112 18450 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46443
I20260923 21:05:26.828478 18450 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:26.829293 18450 heartbeater.cc:511] Master 127.0.97.62:46443 requested a full tablet report, sending...
I20260923 21:05:26.831039 18133 ts_manager.cc:194] Registered new tserver with Master: e3fe6a31d4b74c98a3f758fb0a74005c (127.0.97.2:38997)
I20260923 21:05:26.832340 18133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:47679
W20260923 21:05:27.175005 18453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:27.175307 18453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:27.175395 18453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:27.182888 18453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:27.183000 18453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:27.183108 18453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:05:27.192695 18453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:27.194150 18453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:27.195950 18453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:27.210114 18461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:27.210436 18458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:27.210143 18459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:27.216285 18453 server_base.cc:1034] running on GCE node
I20260923 21:05:27.216989 18453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:27.218182 18453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:27.219444 18453 hybrid_clock.cc:648] HybridClock initialized: now 1790197527219396 us; error 71 us; skew 500 ppm
I20260923 21:05:27.219842 18453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:27.222256 18453 webserver.cc:460] Webserver started at http://127.0.97.3:34555/ using document root <none> and password file <none>
I20260923 21:05:27.223048 18453 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:27.223192 18453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:27.223531 18453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:27.226636 18453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "85f4e240ec484d6db30d4bb10c904033"
format_stamp: "Formatted at 2026-09-23 21:05:27 on dist-test-slave-8xn5"
server_key: "19abce64b56232454382d81c45a30bce"
server_key_iv: "b7bf4375d7d60ba62a1ef16e37b4dc4a"
server_key_version: "encryptionkey@0"
I20260923 21:05:27.227505 18453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "85f4e240ec484d6db30d4bb10c904033"
format_stamp: "Formatted at 2026-09-23 21:05:27 on dist-test-slave-8xn5"
server_key: "19abce64b56232454382d81c45a30bce"
server_key_iv: "b7bf4375d7d60ba62a1ef16e37b4dc4a"
server_key_version: "encryptionkey@0"
I20260923 21:05:27.232885 18453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:05:27.236627 18467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:27.238163 18453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 21:05:27.238402 18453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "85f4e240ec484d6db30d4bb10c904033"
format_stamp: "Formatted at 2026-09-23 21:05:27 on dist-test-slave-8xn5"
server_key: "19abce64b56232454382d81c45a30bce"
server_key_iv: "b7bf4375d7d60ba62a1ef16e37b4dc4a"
server_key_version: "encryptionkey@0"
I20260923 21:05:27.238644 18453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:27.259001 18453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:27.259979 18453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:27.260300 18453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:27.261391 18453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:27.263068 18453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:27.263206 18453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:27.263331 18453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:27.263422 18453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:27.298799 18453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45349
I20260923 21:05:27.298889 18579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45349 every 8 connection(s)
I20260923 21:05:27.300138 18453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:05:27.306010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 18453
I20260923 21:05:27.306211 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:05:27.306710 388 external_mini_cluster.cc:1428] Setting key 3381e44e9f48186f69a8f2366f8921e4
I20260923 21:05:27.313686 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.302857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39547 (local address 127.0.97.62:46443)
0923 21:05:27.303046 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.303051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.304436 (+ 1385us) server_negotiation.cc:407] Connection header received
0923 21:05:27.306420 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.306428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.306534 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.306654 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:27.308107 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.309143 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.311001 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.311412 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.311512 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.312943 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:27.312951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:27.312958 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:27.312987 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:27.313019 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:27.313031 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:27.313104 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.313314 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.313393 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:05:27.314826 18582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.303378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46443 (local address 127.0.97.3:39547)
0923 21:05:27.304276 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:27.304312 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:27.306045 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:27.306923 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:27.306932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:27.307028 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:27.307829 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.307858 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.309311 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.309316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.310858 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.310870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.311666 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.311671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.311868 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.312610 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:27.312613 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:27.312655 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:27.312743 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:27.313131 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:27.313138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:27.313145 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 21:05:27.313290 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 21:05:27.313682 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":282,"threads_started":1}
I20260923 21:05:27.316365 18580 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46443
I20260923 21:05:27.316649 18580 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:27.317385 18580 heartbeater.cc:511] Master 127.0.97.62:46443 requested a full tablet report, sending...
I20260923 21:05:27.318939 18133 ts_manager.cc:194] Registered new tserver with Master: 85f4e240ec484d6db30d4bb10c904033 (127.0.97.3:45349)
I20260923 21:05:27.319823 18133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39547
I20260923 21:05:27.322203 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:05:27.332816 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.326183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.0.97.62:46443)
0923 21:05:27.326477 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.326486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.327147 (+ 661us) server_negotiation.cc:407] Connection header received
0923 21:05:27.327355 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.327363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.327452 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.327548 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:27.328177 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.328992 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.330504 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.330899 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.331002 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.332053 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:27.332057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:27.332060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:27.332079 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:27.332112 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:27.332124 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:27.332203 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.332297 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.332374 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 21:05:27.337229 18133 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:05:27.344226 18133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:05:27.361845 18319 heartbeater.cc:503] Master 127.0.97.62:46443 was elected leader, sending a full tablet report...
I20260923 21:05:27.369874 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.358303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.0.97.1:35781)
0923 21:05:27.360950 (+ 2647us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.360959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.360990 (+ 31us) server_negotiation.cc:407] Connection header received
0923 21:05:27.361082 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.361090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.361331 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.362217 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:27.362927 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.364330 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.367449 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.368501 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.368668 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.368987 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.369250 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.369419 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2465,"thread_start_us":2368,"threads_started":1}
I20260923 21:05:27.370007 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.357348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45349 (local address 127.0.0.1:52220)
0923 21:05:27.358391 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:27.358449 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:27.358663 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:27.360821 (+ 2158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:27.360828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:27.360859 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:27.361316 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.361329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.364228 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.364233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.366948 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.366967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.369045 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.369052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.369242 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.369278 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 21:05:27.369349 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 21:05:27.369513 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":128,"thread_start_us":451,"threads_started":1}
I20260923 21:05:27.370848 18593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.357598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45734)
0923 21:05:27.359335 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:27.359363 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:27.359484 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:27.361558 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:27.361562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:27.361591 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:27.361944 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.361956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.364476 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.364484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.367145 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.367169 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.368734 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.368741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.368971 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.369041 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 21:05:27.369200 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 21:05:27.369402 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1555,"mutex_wait_us":81,"thread_start_us":2368,"threads_started":1}
I20260923 21:05:27.371338 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.357802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.0.97.3:45349)
0923 21:05:27.360058 (+ 2256us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.360077 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.360113 (+ 36us) server_negotiation.cc:407] Connection header received
0923 21:05:27.360214 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.360221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.360495 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.361527 (+ 1032us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:27.362425 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.364098 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.367252 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.368812 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.369715 (+ 903us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.370175 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.370247 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.370467 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2080,"thread_start_us":1930,"threads_started":1}
I20260923 21:05:27.371803 18596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.359429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38997 (local address 127.0.0.1:47456)
0923 21:05:27.361394 (+ 1965us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:27.361424 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:27.361565 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:27.362748 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:27.362754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:27.362778 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:27.363944 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.363957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.366158 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.366164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.368619 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.368635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.370990 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.370997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.371191 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.371222 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 21:05:27.371361 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 21:05:27.371549 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1791,"mutex_wait_us":27,"thread_start_us":1634,"threads_started":1}
I20260923 21:05:27.372726 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.360010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.0.97.2:38997)
0923 21:05:27.362117 (+ 2107us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.362124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.362153 (+ 29us) server_negotiation.cc:407] Connection header received
0923 21:05:27.362261 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.362268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.362519 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.364131 (+ 1612us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:27.364369 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.365967 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.368929 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.370744 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.371135 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.372107 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.372219 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.372395 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1959,"thread_start_us":1835,"threads_started":1}
I20260923 21:05:27.375095 18254 tablet_service.cc:1467] Processing CreateTablet for tablet 72334cce14084174a852292b50753433 (DEFAULT_TABLE table=test-table [id=3941a1eae9f74f1baab3f15781d3394a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:27.375348 18515 tablet_service.cc:1467] Processing CreateTablet for tablet 72334cce14084174a852292b50753433 (DEFAULT_TABLE table=test-table [id=3941a1eae9f74f1baab3f15781d3394a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:27.376446 18254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72334cce14084174a852292b50753433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:27.376914 18515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72334cce14084174a852292b50753433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:27.376652 18385 tablet_service.cc:1467] Processing CreateTablet for tablet 72334cce14084174a852292b50753433 (DEFAULT_TABLE table=test-table [id=3941a1eae9f74f1baab3f15781d3394a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:27.378077 18385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72334cce14084174a852292b50753433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:27.388589 18598 tablet_bootstrap.cc:492] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Bootstrap starting.
I20260923 21:05:27.393085 18599 tablet_bootstrap.cc:492] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Bootstrap starting.
I20260923 21:05:27.393453 18598 tablet_bootstrap.cc:654] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:27.395130 18600 tablet_bootstrap.cc:492] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Bootstrap starting.
I20260923 21:05:27.395105 18598 log.cc:824] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:27.397074 18599 tablet_bootstrap.cc:654] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:27.398491 18599 log.cc:824] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:27.399026 18600 tablet_bootstrap.cc:654] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:27.399497 18598 tablet_bootstrap.cc:492] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: No bootstrap required, opened a new log
I20260923 21:05:27.399839 18598 ts_tablet_manager.cc:1397] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20260923 21:05:27.400460 18600 log.cc:824] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:27.402482 18599 tablet_bootstrap.cc:492] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: No bootstrap required, opened a new log
I20260923 21:05:27.402786 18599 ts_tablet_manager.cc:1397] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260923 21:05:27.406065 18600 tablet_bootstrap.cc:492] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: No bootstrap required, opened a new log
I20260923 21:05:27.406446 18600 ts_tablet_manager.cc:1397] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260923 21:05:27.408267 18598 raft_consensus.cc:348] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.408674 18598 raft_consensus.cc:374] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:27.408767 18598 raft_consensus.cc:729] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40aea24bc9fc4ac995cf76167aa0fca6, State: Initialized, Role: FOLLOWER
I20260923 21:05:27.409437 18598 consensus_queue.cc:260] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.411320 18598 ts_tablet_manager.cc:1428] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260923 21:05:27.415158 18599 raft_consensus.cc:348] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.415693 18599 raft_consensus.cc:374] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:27.415807 18599 raft_consensus.cc:729] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f4e240ec484d6db30d4bb10c904033, State: Initialized, Role: FOLLOWER
I20260923 21:05:27.416694 18599 consensus_queue.cc:260] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.417873 18580 heartbeater.cc:503] Master 127.0.97.62:46443 was elected leader, sending a full tablet report...
I20260923 21:05:27.418720 18599 ts_tablet_manager.cc:1428] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260923 21:05:27.418622 18600 raft_consensus.cc:348] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.419138 18600 raft_consensus.cc:374] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:27.419246 18600 raft_consensus.cc:729] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fe6a31d4b74c98a3f758fb0a74005c, State: Initialized, Role: FOLLOWER
I20260923 21:05:27.419812 18600 consensus_queue.cc:260] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.420922 18450 heartbeater.cc:503] Master 127.0.97.62:46443 was elected leader, sending a full tablet report...
I20260923 21:05:27.421502 18600 ts_tablet_manager.cc:1428] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
W20260923 21:05:27.551939 18581 tablet.cc:2369] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:27.563326 18451 tablet.cc:2369] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:27.587870 18320 tablet.cc:2369] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:27.711062 18606 raft_consensus.cc:482] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:27.711479 18606 raft_consensus.cc:504] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.713440 18606 leader_election.cc:288] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85f4e240ec484d6db30d4bb10c904033 (127.0.97.3:45349), 40aea24bc9fc4ac995cf76167aa0fca6 (127.0.97.1:35781)
I20260923 21:05:27.720775 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.713819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43871 (local address 127.0.97.3:45349)
0923 21:05:27.714062 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.714071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.714962 (+ 891us) server_negotiation.cc:407] Connection header received
0923 21:05:27.715163 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.715169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.715255 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.715346 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:27.716032 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.716800 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.718988 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.719757 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.719902 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.720010 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.720396 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.720522 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 21:05:27.720894 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.713724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45349 (local address 127.0.97.2:43871)
0923 21:05:27.714833 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:27.714862 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:27.715016 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:27.715376 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:27.715383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:27.715410 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:27.715855 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.715866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.717016 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.717044 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.718786 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.718798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.719959 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.719965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.720198 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.720223 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 21:05:27.720353 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 21:05:27.720491 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":117,"spinlock_wait_cycles":29952,"thread_start_us":531,"threads_started":1}
I20260923 21:05:27.720947 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.714099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59661 (local address 127.0.97.1:35781)
0923 21:05:27.714362 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.714368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.715162 (+ 794us) server_negotiation.cc:407] Connection header received
0923 21:05:27.715327 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.715334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.715425 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.715532 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:27.716233 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.716941 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.719110 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.719924 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.720085 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.720187 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.720537 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.720678 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 21:05:27.721627 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.713905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.97.2:59661)
0923 21:05:27.715009 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:27.715025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:27.715179 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:27.715565 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:27.715568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:27.715607 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:27.716059 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.716072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.717115 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.717119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.718789 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:27.718800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.720126 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.720134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:27.720309 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.720364 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 21:05:27.721148 (+ 784us) client_negotiation.cc:240] Negotiation successful
0923 21:05:27.721293 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":90,"spinlock_wait_cycles":1280,"thread_start_us":767,"threads_started":1}
I20260923 21:05:27.721899 18535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72334cce14084174a852292b50753433" candidate_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f4e240ec484d6db30d4bb10c904033" is_pre_election: true
I20260923 21:05:27.722537 18535 raft_consensus.cc:2454] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fe6a31d4b74c98a3f758fb0a74005c in term 0.
I20260923 21:05:27.722792 18274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72334cce14084174a852292b50753433" candidate_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" is_pre_election: true
I20260923 21:05:27.723237 18338 leader_election.cc:302] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f4e240ec484d6db30d4bb10c904033, e3fe6a31d4b74c98a3f758fb0a74005c; no voters:
I20260923 21:05:27.723408 18274 raft_consensus.cc:2454] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fe6a31d4b74c98a3f758fb0a74005c in term 0.
I20260923 21:05:27.723657 18606 raft_consensus.cc:2781] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:05:27.723798 18606 raft_consensus.cc:482] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:05:27.723915 18606 raft_consensus.cc:3037] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:27.726264 18606 raft_consensus.cc:504] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.726985 18606 leader_election.cc:288] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [CANDIDATE]: Term 1 election: Requested vote from peers 85f4e240ec484d6db30d4bb10c904033 (127.0.97.3:45349), 40aea24bc9fc4ac995cf76167aa0fca6 (127.0.97.1:35781)
I20260923 21:05:27.727442 18535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72334cce14084174a852292b50753433" candidate_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85f4e240ec484d6db30d4bb10c904033"
I20260923 21:05:27.727552 18274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72334cce14084174a852292b50753433" candidate_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40aea24bc9fc4ac995cf76167aa0fca6"
I20260923 21:05:27.727672 18535 raft_consensus.cc:3037] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:27.727782 18274 raft_consensus.cc:3037] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:27.731173 18535 raft_consensus.cc:2454] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fe6a31d4b74c98a3f758fb0a74005c in term 1.
I20260923 21:05:27.731173 18274 raft_consensus.cc:2454] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fe6a31d4b74c98a3f758fb0a74005c in term 1.
I20260923 21:05:27.731652 18338 leader_election.cc:302] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85f4e240ec484d6db30d4bb10c904033, e3fe6a31d4b74c98a3f758fb0a74005c; no voters:
I20260923 21:05:27.731950 18606 raft_consensus.cc:2781] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:27.732492 18606 raft_consensus.cc:686] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 1 LEADER]: Becoming Leader. State: Replica: e3fe6a31d4b74c98a3f758fb0a74005c, State: Running, Role: LEADER
I20260923 21:05:27.733072 18606 consensus_queue.cc:237] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } }
I20260923 21:05:27.738288 18132 catalog_manager.cc:5495] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fe6a31d4b74c98a3f758fb0a74005c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3fe6a31d4b74c98a3f758fb0a74005c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38997 } health_report { overall_health: HEALTHY } } }
I20260923 21:05:27.833129 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.827665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.0.97.2:38997)
0923 21:05:27.827864 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.827870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.827890 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:05:27.828063 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.828067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.828156 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.828311 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:05:27.828695 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.829411 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.830895 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.831319 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.831461 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.831999 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:05:27.832511 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:05:27.832683 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.832740 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.832834 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 21:05:27.841831 18535 raft_consensus.cc:1261] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 1 FOLLOWER]: Refusing update from remote peer e3fe6a31d4b74c98a3f758fb0a74005c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:27.841831 18274 raft_consensus.cc:1261] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 1 FOLLOWER]: Refusing update from remote peer e3fe6a31d4b74c98a3f758fb0a74005c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:27.842895 18609 consensus_queue.cc:1035] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40aea24bc9fc4ac995cf76167aa0fca6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:27.843312 18606 consensus_queue.cc:1035] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85f4e240ec484d6db30d4bb10c904033" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:27.855280 18612 mvcc.cc:205] Tried to move back new op lower bound from 7332649074029735936 to 7332649073599422464. Current Snapshot: MvccSnapshot[applied={T|T < 7332649074029735936}]
I20260923 21:05:27.871632 18133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:27.871932 18133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:27.875838 18133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 173feef1b1fd4f398d9a85c79225682d: Sending DeleteTablet for 3 replicas of tablet 72334cce14084174a852292b50753433
I20260923 21:05:27.877241 18254 tablet_service.cc:1514] Processing DeleteTablet for tablet 72334cce14084174a852292b50753433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:27 UTC) from {username='slave'} at 127.0.0.1:45734
I20260923 21:05:27.877275 18515 tablet_service.cc:1514] Processing DeleteTablet for tablet 72334cce14084174a852292b50753433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:27 UTC) from {username='slave'} at 127.0.0.1:52220
I20260923 21:05:27.877516 18385 tablet_service.cc:1514] Processing DeleteTablet for tablet 72334cce14084174a852292b50753433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:27 UTC) from {username='slave'} at 127.0.0.1:47456
I20260923 21:05:27.878274 18622 tablet_replica.cc:330] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: stopping tablet replica
I20260923 21:05:27.878973 18622 raft_consensus.cc:2229] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:27.879025 18624 tablet_replica.cc:330] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: stopping tablet replica
I20260923 21:05:27.879427 18624 raft_consensus.cc:2229] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:05:27.879515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 18192
I20260923 21:05:27.879724 18622 raft_consensus.cc:2258] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:27.880122 18624 raft_consensus.cc:2258] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:27.882117 18622 ts_tablet_manager.cc:1905] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:27.882123 18624 ts_tablet_manager.cc:1905] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:27.889925 18624 ts_tablet_manager.cc:1918] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:27.890125 18624 log.cc:1196] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/72334cce14084174a852292b50753433
I20260923 21:05:27.890702 18624 ts_tablet_manager.cc:1939] T 72334cce14084174a852292b50753433 P e3fe6a31d4b74c98a3f758fb0a74005c: Deleting consensus metadata
I20260923 21:05:27.891507 18622 ts_tablet_manager.cc:1918] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:27.891810 18622 log.cc:1196] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/72334cce14084174a852292b50753433
I20260923 21:05:27.892416 18622 ts_tablet_manager.cc:1939] T 72334cce14084174a852292b50753433 P 85f4e240ec484d6db30d4bb10c904033: Deleting consensus metadata
I20260923 21:05:27.892733 18117 catalog_manager.cc:4841] TS e3fe6a31d4b74c98a3f758fb0a74005c (127.0.97.2:38997): tablet 72334cce14084174a852292b50753433 (table test-table [id=3941a1eae9f74f1baab3f15781d3394a]) successfully deleted
I20260923 21:05:27.894490 18117 catalog_manager.cc:4841] TS 85f4e240ec484d6db30d4bb10c904033 (127.0.97.3:45349): tablet 72334cce14084174a852292b50753433 (table test-table [id=3941a1eae9f74f1baab3f15781d3394a]) successfully deleted
I20260923 21:05:27.898773 18623 tablet_replica.cc:330] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: stopping tablet replica
I20260923 21:05:27.899207 18623 raft_consensus.cc:2229] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:27.899811 18623 raft_consensus.cc:2258] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:27.901582 18623 ts_tablet_manager.cc:1905] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:27.906656 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:27.882063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.0.97.1:35781)
0923 21:05:27.882265 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:27.882271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:27.898236 (+ 15965us) server_negotiation.cc:407] Connection header received
0923 21:05:27.898522 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:27.898528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:27.898648 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:27.898733 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:27.899485 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.900644 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.902666 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:27.903357 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:27.903497 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:27.904837 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:27.904850 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:27.904869 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:27.905697 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:27.905764 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:27.905788 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:27.905924 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:27.906109 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 21:05:27.906244 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:05:27.907508 18314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:27.997298 18623 ts_tablet_manager.cc:1918] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:27.997620 18623 log.cc:1196] T 72334cce14084174a852292b50753433 P 40aea24bc9fc4ac995cf76167aa0fca6: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/72334cce14084174a852292b50753433
I20260923 21:05:27.997961 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 18192
I20260923 21:05:28.016470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 18323
W20260923 21:05:28.016464 18119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35781 (error 108)
I20260923 21:05:28.018790 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.017841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45744)
0923 21:05:28.018388 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.018545 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
W20260923 21:05:28.019174 18119 catalog_manager.cc:4584] TS 40aea24bc9fc4ac995cf76167aa0fca6 (127.0.97.1:35781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72334cce14084174a852292b50753433: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
I20260923 21:05:28.025501 18597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.017363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.0.97.2:38997)
0923 21:05:28.017565 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:28.017572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:28.017988 (+ 416us) server_negotiation.cc:407] Connection header received
0923 21:05:28.018249 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:28.018255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:28.018407 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:28.018493 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:28.019244 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:28.020430 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:28.021751 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:28.022533 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:28.022665 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:28.023725 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:28.023737 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:28.023759 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:28.024594 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:28.024662 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:28.024687 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:28.024799 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:28.024987 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 21:05:28.025099 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 21:05:28.026271 18445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:28.072837 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.072349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45750)
0923 21:05:28.072591 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.072692 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 21:05:28.074209 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.073640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45758)
0923 21:05:28.073981 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.074064 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 21:05:28.135079 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.134542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45764)
0923 21:05:28.134780 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.134920 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 21:05:28.136224 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.135825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45766)
0923 21:05:28.135997 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.136088 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 21:05:28.143237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 18323
I20260923 21:05:28.162487 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 18453
I20260923 21:05:28.169358 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.163384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.0.97.3:45349)
0923 21:05:28.163551 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:28.163555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:28.163573 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:05:28.163689 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:28.163693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:28.163773 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:28.163858 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:28.164271 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:28.165008 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:28.166398 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:28.166794 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:28.166887 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:28.167905 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:28.167913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:28.167929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:28.168548 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:28.168598 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:28.168619 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:28.168706 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:28.168851 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 21:05:28.168941 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 21:05:28.170080 18575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:28.217257 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.216601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45782)
0923 21:05:28.216873 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.217053 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 21:05:28.218595 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.218058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45790)
0923 21:05:28.218317 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.218440 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 21:05:28.264871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 18453
I20260923 21:05:28.281728 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 18102
I20260923 21:05:28.283133 18163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:28.398175 18628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:28.397550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35781 (local address 127.0.0.1:45796)
0923 21:05:28.397743 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:28.397942 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 21:05:28.398522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 18102
2026-09-23T21:05:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3221 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:28 dist-test-slave-8xn5 krb5kdc[18634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:28 dist-test-slave-8xn5 krb5kdc[18634](info): set up 2 sockets
Sep 23 21:05:28 dist-test-slave-8xn5 krb5kdc[18634](info): commencing operation
krb5kdc: starting...
W20260923 21:05:29.518611 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:29 dist-test-slave-8xn5 krb5kdc[18634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:05:29.785808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34647
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:30.125897 18647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:30.126291 18647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:30.126483 18647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:30.133816 18647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:30.133980 18647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:30.134094 18647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:30.134205 18647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:30.134294 18647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:30.144411 18647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:30.145788 18647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:30.147154 18647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:30.159379 18652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:30.159307 18655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:30.159420 18653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:30.160681 18647 server_base.cc:1034] running on GCE node
I20260923 21:05:30.161525 18647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:30.163177 18647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:30.164389 18647 hybrid_clock.cc:648] HybridClock initialized: now 1790197530164332 us; error 48 us; skew 500 ppm
I20260923 21:05:30.164767 18647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:30 dist-test-slave-8xn5 krb5kdc[18634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:30.168066 18647 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:05:30.169903 18647 webserver.cc:460] Webserver started at http://127.0.97.62:41607/ using document root <none> and password file <none>
I20260923 21:05:30.170652 18647 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:30.170758 18647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:30.171033 18647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:30.174003 18647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "1f50f7db78934090abaa9b51238347d6"
format_stamp: "Formatted at 2026-09-23 21:05:30 on dist-test-slave-8xn5"
server_key: "ad4be5c514b304086ee6e24d60685ff3"
server_key_iv: "1db0d4b7facd7f1c6e9e9593a97649b7"
server_key_version: "encryptionkey@0"
I20260923 21:05:30.174854 18647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "1f50f7db78934090abaa9b51238347d6"
format_stamp: "Formatted at 2026-09-23 21:05:30 on dist-test-slave-8xn5"
server_key: "ad4be5c514b304086ee6e24d60685ff3"
server_key_iv: "1db0d4b7facd7f1c6e9e9593a97649b7"
server_key_version: "encryptionkey@0"
I20260923 21:05:30.180263 18647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:05:30.183965 18662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:30.185392 18647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:05:30.185551 18647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "1f50f7db78934090abaa9b51238347d6"
format_stamp: "Formatted at 2026-09-23 21:05:30 on dist-test-slave-8xn5"
server_key: "ad4be5c514b304086ee6e24d60685ff3"
server_key_iv: "1db0d4b7facd7f1c6e9e9593a97649b7"
server_key_version: "encryptionkey@0"
I20260923 21:05:30.185729 18647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:30.202831 18647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:30.207324 18647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:30.207574 18647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:30.229661 18647 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34647
I20260923 21:05:30.229699 18713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34647 every 8 connection(s)
I20260923 21:05:30.231168 18647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:30.235497 18714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:30.240906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 18647
I20260923 21:05:30.241120 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:30.241595 388 external_mini_cluster.cc:1428] Setting key 8761cfef3e992e2244ccc8674a4275d9
I20260923 21:05:30.248569 18714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6: Bootstrap starting.
I20260923 21:05:30.252339 18714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:30.253502 18714 log.cc:824] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:30.256546 18714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6: No bootstrap required, opened a new log
Sep 23 21:05:30 dist-test-slave-8xn5 krb5kdc[18634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:30.264864 18714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f50f7db78934090abaa9b51238347d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260923 21:05:30.265306 18714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:30.265455 18714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f50f7db78934090abaa9b51238347d6, State: Initialized, Role: FOLLOWER
I20260923 21:05:30.266158 18714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f50f7db78934090abaa9b51238347d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260923 21:05:30.266464 18714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:30.266623 18714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:30.266840 18714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:30.267053 18717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:30.243579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.0.97.62:34647)
0923 21:05:30.244433 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:30.244456 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:30.244510 (+ 54us) server_negotiation.cc:407] Connection header received
0923 21:05:30.247031 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:30.247062 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:30.247582 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:30.248043 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:30.249526 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:30.251583 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:30.253117 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:30.254218 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:30.254468 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:30.258851 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:30.258859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:30.258878 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:30.258925 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:30.262377 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:30.263168 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:30.263176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:30.263183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:30.263370 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:30.263782 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:30.263793 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:30.263801 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:30.264381 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:30.264702 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:30.265169 (+ 467us) server_negotiation.cc:299] Negotiation successful
0923 21:05:30.265677 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":369,"threads_started":1}
I20260923 21:05:30.269867 18714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f50f7db78934090abaa9b51238347d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260923 21:05:30.270484 18714 leader_election.cc:302] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f50f7db78934090abaa9b51238347d6; no voters:
I20260923 21:05:30.271112 18714 leader_election.cc:288] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:30.271222 18719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:30.271859 18719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [term 1 LEADER]: Becoming Leader. State: Replica: 1f50f7db78934090abaa9b51238347d6, State: Running, Role: LEADER
I20260923 21:05:30.272496 18719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f50f7db78934090abaa9b51238347d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } }
I20260923 21:05:30.273424 18714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:30.275629 18721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f50f7db78934090abaa9b51238347d6. Latest consensus state: current_term: 1 leader_uuid: "1f50f7db78934090abaa9b51238347d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f50f7db78934090abaa9b51238347d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } } }
I20260923 21:05:30.276249 18721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:30.276628 18720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f50f7db78934090abaa9b51238347d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f50f7db78934090abaa9b51238347d6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34647 } } }
I20260923 21:05:30.277063 18720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:30.277894 18726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:30.284078 18726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:30.294149 18726 catalog_manager.cc:1347] Generated new cluster ID: ad9c402350514cd1b2cf267d8b75be09
I20260923 21:05:30.294314 18726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:30.306074 18726 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:30.308182 18726 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:30.320462 18726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6: Generated new TSK 0
I20260923 21:05:30.321403 18726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:05:30.402716 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:30.747079 18740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:30.747380 18740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:30.747514 18740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:30.755142 18740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:30.755249 18740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:30.755465 18740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:05:30.765535 18740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:30.767067 18740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:30.768426 18740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:30.783049 18745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:30.783000 18746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:30.783027 18748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:30.784520 18740 server_base.cc:1034] running on GCE node
I20260923 21:05:30.785285 18740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:30.786309 18740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:30.787654 18740 hybrid_clock.cc:648] HybridClock initialized: now 1790197530787607 us; error 36 us; skew 500 ppm
I20260923 21:05:30.787999 18740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:30 dist-test-slave-8xn5 krb5kdc[18634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:30.795589 18740 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:05:30.797329 18740 webserver.cc:460] Webserver started at http://127.0.97.1:36671/ using document root <none> and password file <none>
I20260923 21:05:30.798086 18740 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:30.798235 18740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:30.798614 18740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:30.801644 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "382a454661834f3e9ff6e0e114b7b10e"
format_stamp: "Formatted at 2026-09-23 21:05:30 on dist-test-slave-8xn5"
server_key: "34f7e65bc5795708e01b70bfc690d037"
server_key_iv: "d433b031f8a718052547959764ca5f50"
server_key_version: "encryptionkey@0"
I20260923 21:05:30.802567 18740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "382a454661834f3e9ff6e0e114b7b10e"
format_stamp: "Formatted at 2026-09-23 21:05:30 on dist-test-slave-8xn5"
server_key: "34f7e65bc5795708e01b70bfc690d037"
server_key_iv: "d433b031f8a718052547959764ca5f50"
server_key_version: "encryptionkey@0"
I20260923 21:05:30.808187 18740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:05:30.812156 18755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:30.813838 18740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:05:30.814072 18740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "382a454661834f3e9ff6e0e114b7b10e"
format_stamp: "Formatted at 2026-09-23 21:05:30 on dist-test-slave-8xn5"
server_key: "34f7e65bc5795708e01b70bfc690d037"
server_key_iv: "d433b031f8a718052547959764ca5f50"
server_key_version: "encryptionkey@0"
I20260923 21:05:30.814527 18740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:30.827967 18740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:30.832880 18740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:30.833206 18740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:30.834213 18740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:30.835724 18740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:30.835827 18740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:30.835974 18740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:30.836045 18740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:30.872915 18740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44309
I20260923 21:05:30.872974 18867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44309 every 8 connection(s)
I20260923 21:05:30.874550 18740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:05:30.879038 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 18740
I20260923 21:05:30.879225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:05:30.879763 388 external_mini_cluster.cc:1428] Setting key 1eddcc71ef537d22ca315a95ecbafa1d
Sep 23 21:05:30 dist-test-slave-8xn5 krb5kdc[18634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:30.897939 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:30.877321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54307 (local address 127.0.97.62:34647)
0923 21:05:30.877790 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:30.877796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:30.878978 (+ 1182us) server_negotiation.cc:407] Connection header received
0923 21:05:30.881326 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:30.881332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:30.881526 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:30.881639 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:30.883595 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:30.884788 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:30.886652 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:30.887422 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:30.887578 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:30.891451 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:30.891458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:30.891462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:30.891503 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:30.894212 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:30.895419 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:30.895426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:30.895429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:30.895569 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:30.896098 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:30.896101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:30.896103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:30.896374 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:30.896491 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:30.897463 (+ 972us) server_negotiation.cc:299] Negotiation successful
0923 21:05:30.897652 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":215,"threads_started":1}
I20260923 21:05:30.899664 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:30.877900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34647 (local address 127.0.97.1:54307)
0923 21:05:30.878784 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:30.878826 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:30.881028 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:30.881975 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:30.881985 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:30.882749 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:30.883381 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:30.883402 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:30.884987 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:30.884991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:30.886485 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:30.886503 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:30.887647 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:30.887652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:30.887867 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:30.888601 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:30.888626 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:30.891146 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:30.894434 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:30.894448 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:30.894472 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:30.895258 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:30.895688 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:30.895696 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:30.895702 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:30.895984 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:30.896507 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:30.896519 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:30.897035 (+ 516us) client_negotiation.cc:769] Sending connection context
0923 21:05:30.897366 (+ 331us) client_negotiation.cc:240] Negotiation successful
0923 21:05:30.898028 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":401,"threads_started":1}
I20260923 21:05:30.901448 18868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34647
I20260923 21:05:30.901799 18868 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:30.902727 18868 heartbeater.cc:511] Master 127.0.97.62:34647 requested a full tablet report, sending...
I20260923 21:05:30.905267 18679 ts_manager.cc:194] Registered new tserver with Master: 382a454661834f3e9ff6e0e114b7b10e (127.0.97.1:44309)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:05:30.907090 18679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:05:30.942807 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:31.300112 18874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:31.300482 18874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:31.300659 18874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:31.308188 18874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:31.308370 18874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:31.308564 18874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:05:31.318763 18874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:31.320218 18874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:31.321589 18874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:31.336719 18879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:31.336709 18882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:31.336755 18880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:31.338250 18874 server_base.cc:1034] running on GCE node
I20260923 21:05:31.338984 18874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:31.340036 18874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:31.341306 18874 hybrid_clock.cc:648] HybridClock initialized: now 1790197531341220 us; error 50 us; skew 500 ppm
I20260923 21:05:31.341670 18874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:31 dist-test-slave-8xn5 krb5kdc[18634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:31.346247 18874 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:05:31.347962 18874 webserver.cc:460] Webserver started at http://127.0.97.2:39355/ using document root <none> and password file <none>
I20260923 21:05:31.348726 18874 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:31.348870 18874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:31.349217 18874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:31.352506 18874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "66a009cf596b46c8b1a7e5d0325193c7"
format_stamp: "Formatted at 2026-09-23 21:05:31 on dist-test-slave-8xn5"
server_key: "29a5b99c71c3776903cc2af9c5e8a898"
server_key_iv: "ea61f1d6203ad9a85f2d027b69ed923d"
server_key_version: "encryptionkey@0"
I20260923 21:05:31.353376 18874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "66a009cf596b46c8b1a7e5d0325193c7"
format_stamp: "Formatted at 2026-09-23 21:05:31 on dist-test-slave-8xn5"
server_key: "29a5b99c71c3776903cc2af9c5e8a898"
server_key_iv: "ea61f1d6203ad9a85f2d027b69ed923d"
server_key_version: "encryptionkey@0"
I20260923 21:05:31.358804 18874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 21:05:31.362797 18889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:31.364217 18874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:05:31.364408 18874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "66a009cf596b46c8b1a7e5d0325193c7"
format_stamp: "Formatted at 2026-09-23 21:05:31 on dist-test-slave-8xn5"
server_key: "29a5b99c71c3776903cc2af9c5e8a898"
server_key_iv: "ea61f1d6203ad9a85f2d027b69ed923d"
server_key_version: "encryptionkey@0"
I20260923 21:05:31.364617 18874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:31.384213 18874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:31.388975 18874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:31.389343 18874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:31.390427 18874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:31.391857 18874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:31.391965 18874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:31.392125 18874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:31.392220 18874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:31.428361 18874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43573
I20260923 21:05:31.428475 19001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43573 every 8 connection(s)
I20260923 21:05:31.429720 18874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:05:31.439993 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 18874
I20260923 21:05:31.440234 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:05:31.440855 388 external_mini_cluster.cc:1428] Setting key 038f93b65be95d4329e600d3efc282b2
Sep 23 21:05:31 dist-test-slave-8xn5 krb5kdc[18634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:31.455022 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:31.432635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49637 (local address 127.0.97.62:34647)
0923 21:05:31.433132 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:31.433138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:31.434197 (+ 1059us) server_negotiation.cc:407] Connection header received
0923 21:05:31.436356 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:31.436364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:31.436469 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:31.436609 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:31.438368 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:31.439127 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:31.441814 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:31.442406 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:31.442572 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:31.447129 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:31.447142 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:31.447150 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:31.447206 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:31.451410 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:31.452217 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:31.452226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:31.452233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:31.452402 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:31.452861 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:31.452869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:31.452876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:31.453213 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:31.453327 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:31.454531 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0923 21:05:31.454722 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":263,"threads_started":1}
I20260923 21:05:31.456487 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:31.433180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34647 (local address 127.0.97.2:49637)
0923 21:05:31.433975 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:31.434020 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:31.436008 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:31.436892 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:31.436901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:31.437489 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:31.438119 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:31.438139 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:31.439369 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:31.439375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:31.441608 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:31.441625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:31.442682 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:31.442691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:31.442983 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:31.443798 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:31.443835 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:31.446827 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:31.451595 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:31.451603 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:31.451619 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:31.452078 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:31.452539 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:31.452542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:31.452544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:31.452740 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:31.453382 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:31.453392 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:31.453964 (+ 572us) client_negotiation.cc:769] Sending connection context
0923 21:05:31.454345 (+ 381us) client_negotiation.cc:240] Negotiation successful
0923 21:05:31.454964 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":307,"threads_started":1}
I20260923 21:05:31.458127 19002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34647
I20260923 21:05:31.458528 19002 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:31.459306 19002 heartbeater.cc:511] Master 127.0.97.62:34647 requested a full tablet report, sending...
I20260923 21:05:31.461138 18679 ts_manager.cc:194] Registered new tserver with Master: 66a009cf596b46c8b1a7e5d0325193c7 (127.0.97.2:43573)
I20260923 21:05:31.462036 18679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49637
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:05:31.502844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:31.858079 19008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:31.858532 19008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:31.858731 19008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:31.866379 19008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:31.866571 19008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:31.866770 19008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:05:31.878116 19008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:31.879599 19008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:31.881016 19008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:31.895536 19013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:31.895879 19014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:31.895679 19016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:31.896764 19008 server_base.cc:1034] running on GCE node
I20260923 21:05:31.897457 19008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:31.898535 19008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:31.899766 19008 hybrid_clock.cc:648] HybridClock initialized: now 1790197531899720 us; error 36 us; skew 500 ppm
I20260923 21:05:31.900118 19008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:31 dist-test-slave-8xn5 krb5kdc[18634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:31.903668 19008 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:05:31.905306 19008 webserver.cc:460] Webserver started at http://127.0.97.3:40197/ using document root <none> and password file <none>
I20260923 21:05:31.906023 19008 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:31.906126 19008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:31.906481 19008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:31.909529 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "b2a3a6e339e84798ae6a93319cacb649"
format_stamp: "Formatted at 2026-09-23 21:05:31 on dist-test-slave-8xn5"
server_key: "8271306aa601b296008a032343fef1e2"
server_key_iv: "5bb0a86606828907516281143b0e1c4c"
server_key_version: "encryptionkey@0"
I20260923 21:05:31.910287 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b2a3a6e339e84798ae6a93319cacb649"
format_stamp: "Formatted at 2026-09-23 21:05:31 on dist-test-slave-8xn5"
server_key: "8271306aa601b296008a032343fef1e2"
server_key_iv: "5bb0a86606828907516281143b0e1c4c"
server_key_version: "encryptionkey@0"
I20260923 21:05:31.911022 18868 heartbeater.cc:503] Master 127.0.97.62:34647 was elected leader, sending a full tablet report...
I20260923 21:05:31.916275 19008 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:05:31.920195 19023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:31.921669 19008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:05:31.921819 19008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "b2a3a6e339e84798ae6a93319cacb649"
format_stamp: "Formatted at 2026-09-23 21:05:31 on dist-test-slave-8xn5"
server_key: "8271306aa601b296008a032343fef1e2"
server_key_iv: "5bb0a86606828907516281143b0e1c4c"
server_key_version: "encryptionkey@0"
I20260923 21:05:31.921999 19008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:31.958462 19008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:31.963372 19008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:31.963716 19008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:31.964787 19008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:31.966257 19008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:31.966403 19008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 21:05:31.966598 19008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:31.966675 19008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:32.003938 19008 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46447
I20260923 21:05:32.004046 19135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46447 every 8 connection(s)
I20260923 21:05:32.005335 19008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:05:32.011075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19008
I20260923 21:05:32.011271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:05:32.011698 388 external_mini_cluster.cc:1428] Setting key a85b1a408c2b98bc2aa0290969d4dbc8
Sep 23 21:05:32 dist-test-slave-8xn5 krb5kdc[18634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:32.027979 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.008479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43031 (local address 127.0.97.62:34647)
0923 21:05:32.009040 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.009045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.010075 (+ 1030us) server_negotiation.cc:407] Connection header received
0923 21:05:32.012360 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.012368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.012470 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.012620 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:32.014295 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.015170 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.016829 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.017239 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.017374 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.021350 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:32.021356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:32.021359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:32.021396 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:32.024136 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.024925 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.024932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.024936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.025084 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.025828 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.025831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.025833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.026216 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:32.026417 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.027479 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.027698 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":336,"threads_started":1}
I20260923 21:05:32.029074 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.009093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34647 (local address 127.0.97.3:43031)
0923 21:05:32.009881 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:32.009923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:32.012041 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:32.012883 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:32.012892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:32.013510 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:32.014109 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.014128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.015357 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.015361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.016668 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.016681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.017494 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.017501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.017785 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.018597 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:32.018625 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:32.021094 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:32.024320 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:32.024330 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:32.024350 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:32.024795 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:32.025281 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:32.025286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:32.025288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:32.025643 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:32.026469 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:32.026475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:32.026991 (+ 516us) client_negotiation.cc:769] Sending connection context
0923 21:05:32.027354 (+ 363us) client_negotiation.cc:240] Negotiation successful
0923 21:05:32.027814 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":336,"threads_started":1}
I20260923 21:05:32.030527 19136 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34647
I20260923 21:05:32.030884 19136 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:32.031631 19136 heartbeater.cc:511] Master 127.0.97.62:34647 requested a full tablet report, sending...
I20260923 21:05:32.033524 18679 ts_manager.cc:194] Registered new tserver with Master: b2a3a6e339e84798ae6a93319cacb649 (127.0.97.3:46447)
I20260923 21:05:32.034394 18679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43031
I20260923 21:05:32.040005 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:05:32.057951 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.044260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.0.97.62:34647)
0923 21:05:32.044456 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.044461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.044905 (+ 444us) server_negotiation.cc:407] Connection header received
0923 21:05:32.045044 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.045049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.045115 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.045202 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:32.046778 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.047578 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.048944 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.049335 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.049439 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.052241 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:32.052247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:32.052250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:32.052289 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:32.055196 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.055999 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.056004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.056006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.056127 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.056570 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.056574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.056576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.056932 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:32.057129 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.057527 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.057682 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 21:05:32.062107 18679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:05:32.069303 18679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:05:32.095702 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.085102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44309 (local address 127.0.0.1:45940)
0923 21:05:32.085647 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:32.085665 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:32.085760 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:32.087180 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:32.087186 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:32.087211 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:32.087647 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.087661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.090028 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.090036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.092519 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.092536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.094513 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.094520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.094740 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.094797 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 21:05:32.094944 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 21:05:32.095164 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":113,"thread_start_us":1166,"threads_started":1}
I20260923 21:05:32.096098 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.083298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46447 (local address 127.0.0.1:46022)
0923 21:05:32.084246 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:32.084288 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:32.084469 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:32.087808 (+ 3339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:32.087812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:32.087851 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:32.088211 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.088221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.091601 (+ 3380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.091606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.093599 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.093610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.095291 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.095301 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.095469 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.095507 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 21:05:32.095612 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 21:05:32.095763 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":1329,"threads_started":1}
I20260923 21:05:32.096287 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.085796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.0.97.1:44309)
0923 21:05:32.086537 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.086544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.086575 (+ 31us) server_negotiation.cc:407] Connection header received
0923 21:05:32.086671 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.086678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.086946 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.087867 (+ 921us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:32.088082 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.089886 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.092703 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.094050 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.094192 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.095703 (+ 1511us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.095773 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.095946 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":635,"threads_started":1}
I20260923 21:05:32.097113 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.083496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43573 (local address 127.0.0.1:40306)
0923 21:05:32.084888 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:32.084908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:32.085020 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:32.085768 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:32.085775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:32.085813 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:32.086260 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.086275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.089895 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.089901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.093093 (+ 3192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.093109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.096513 (+ 3404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.096520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.096645 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.096666 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 21:05:32.096742 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 21:05:32.096834 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1273,"mutex_wait_us":75,"thread_start_us":356,"threads_started":1}
I20260923 21:05:32.097352 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.084058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.0.97.2:43573)
0923 21:05:32.085054 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.085065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.085110 (+ 45us) server_negotiation.cc:407] Connection header received
0923 21:05:32.085210 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.085216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.085523 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.086510 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:32.086727 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.088364 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.093282 (+ 4918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.094808 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.094950 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.095137 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.096834 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.097004 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":1955,"threads_started":1}
I20260923 21:05:32.098240 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.083542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.97.3:46447)
0923 21:05:32.087189 (+ 3647us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.087198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.087232 (+ 34us) server_negotiation.cc:407] Connection header received
0923 21:05:32.087332 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.087338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.087582 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.088369 (+ 787us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:32.089744 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.091422 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.093786 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.095063 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.095266 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.095496 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.095686 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.095885 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3490,"thread_start_us":3332,"threads_started":1}
I20260923 21:05:32.100771 18937 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9929618d3d4726a54cde382cfc3867 (DEFAULT_TABLE table=test-table [id=b8ce580f98324309842ce495c36bd8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:32.100567 18803 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9929618d3d4726a54cde382cfc3867 (DEFAULT_TABLE table=test-table [id=b8ce580f98324309842ce495c36bd8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:32.101224 19071 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9929618d3d4726a54cde382cfc3867 (DEFAULT_TABLE table=test-table [id=b8ce580f98324309842ce495c36bd8dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:32.102123 18937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9929618d3d4726a54cde382cfc3867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:32.102123 18803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9929618d3d4726a54cde382cfc3867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:32.102546 19071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d9929618d3d4726a54cde382cfc3867. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:32.114785 19155 tablet_bootstrap.cc:492] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Bootstrap starting.
I20260923 21:05:32.117499 19156 tablet_bootstrap.cc:492] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Bootstrap starting.
I20260923 21:05:32.118497 19157 tablet_bootstrap.cc:492] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Bootstrap starting.
I20260923 21:05:32.120012 19155 tablet_bootstrap.cc:654] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:32.121106 19155 log.cc:824] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:32.122752 19156 tablet_bootstrap.cc:654] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:32.122907 19157 tablet_bootstrap.cc:654] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:32.124110 19155 tablet_bootstrap.cc:492] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: No bootstrap required, opened a new log
I20260923 21:05:32.124114 19157 log.cc:824] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:32.124114 19156 log.cc:824] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:32.124414 19155 ts_tablet_manager.cc:1397] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 21:05:32.127559 19157 tablet_bootstrap.cc:492] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: No bootstrap required, opened a new log
I20260923 21:05:32.127796 19156 tablet_bootstrap.cc:492] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: No bootstrap required, opened a new log
I20260923 21:05:32.128032 19157 ts_tablet_manager.cc:1397] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 21:05:32.128121 19156 ts_tablet_manager.cc:1397] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 21:05:32.135666 19157 raft_consensus.cc:348] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.135715 19155 raft_consensus.cc:348] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.136093 19157 raft_consensus.cc:374] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:32.136178 19155 raft_consensus.cc:374] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:32.136210 19157 raft_consensus.cc:729] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a009cf596b46c8b1a7e5d0325193c7, State: Initialized, Role: FOLLOWER
I20260923 21:05:32.136281 19155 raft_consensus.cc:729] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a3a6e339e84798ae6a93319cacb649, State: Initialized, Role: FOLLOWER
I20260923 21:05:32.136889 19157 consensus_queue.cc:260] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.137001 19155 consensus_queue.cc:260] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.138088 19002 heartbeater.cc:503] Master 127.0.97.62:34647 was elected leader, sending a full tablet report...
I20260923 21:05:32.138823 19157 ts_tablet_manager.cc:1428] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260923 21:05:32.139542 19156 raft_consensus.cc:348] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.140092 19156 raft_consensus.cc:374] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:32.140210 19156 raft_consensus.cc:729] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382a454661834f3e9ff6e0e114b7b10e, State: Initialized, Role: FOLLOWER
I20260923 21:05:32.140563 19136 heartbeater.cc:503] Master 127.0.97.62:34647 was elected leader, sending a full tablet report...
I20260923 21:05:32.140961 19156 consensus_queue.cc:260] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.142570 19155 ts_tablet_manager.cc:1428] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Time spent starting tablet: real 0.018s user 0.014s sys 0.005s
I20260923 21:05:32.143590 19156 ts_tablet_manager.cc:1428] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260923 21:05:32.181872 19003 tablet.cc:2369] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:32.257228 19137 tablet.cc:2369] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:32.280553 19162 raft_consensus.cc:482] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:32.280913 19162 raft_consensus.cc:504] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.283054 19162 leader_election.cc:288] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66a009cf596b46c8b1a7e5d0325193c7 (127.0.97.2:43573), 382a454661834f3e9ff6e0e114b7b10e (127.0.97.1:44309)
I20260923 21:05:32.290058 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.283327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44309 (local address 127.0.97.3:42569)
0923 21:05:32.283562 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:32.283588 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:32.283728 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:32.284333 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:32.284337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:32.284360 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:32.284574 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.284584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.286128 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.286133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.287671 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.287685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.289194 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.289200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.289393 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.289429 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 21:05:32.289533 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 21:05:32.289681 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 21:05:32.290082 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.283500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42569 (local address 127.0.97.1:44309)
0923 21:05:32.283695 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.283701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.283721 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:05:32.284104 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.284109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.284198 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.284297 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:32.284755 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.285979 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.287884 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.289041 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.289147 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.289225 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.289588 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.289744 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 21:05:32.291141 19164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.283508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43573 (local address 127.0.97.3:60027)
0923 21:05:32.284331 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:32.284361 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:32.284556 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:32.284991 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:32.284996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:32.285023 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:32.285502 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.285517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.286805 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.286809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.288624 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:32.288639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.290513 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.290522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:32.290690 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.290725 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:05:32.290816 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 21:05:32.290913 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":488,"threads_started":1}
I20260923 21:05:32.291281 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.283755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60027 (local address 127.0.97.2:43573)
0923 21:05:32.283984 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.283992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.284455 (+ 463us) server_negotiation.cc:407] Connection header received
0923 21:05:32.284769 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.284773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.284848 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.284920 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:32.285677 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.286631 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.288792 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.290105 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.290243 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.290386 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.290894 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.291034 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 21:05:32.291254 18823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9929618d3d4726a54cde382cfc3867" candidate_uuid: "b2a3a6e339e84798ae6a93319cacb649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382a454661834f3e9ff6e0e114b7b10e" is_pre_election: true
I20260923 21:05:32.292023 18823 raft_consensus.cc:2454] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a3a6e339e84798ae6a93319cacb649 in term 0.
I20260923 21:05:32.292280 18957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9929618d3d4726a54cde382cfc3867" candidate_uuid: "b2a3a6e339e84798ae6a93319cacb649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a009cf596b46c8b1a7e5d0325193c7" is_pre_election: true
I20260923 21:05:32.292783 19026 leader_election.cc:302] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382a454661834f3e9ff6e0e114b7b10e, b2a3a6e339e84798ae6a93319cacb649; no voters:
I20260923 21:05:32.292832 18957 raft_consensus.cc:2454] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2a3a6e339e84798ae6a93319cacb649 in term 0.
I20260923 21:05:32.293176 19162 raft_consensus.cc:2781] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:05:32.293363 19162 raft_consensus.cc:482] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:05:32.293503 19162 raft_consensus.cc:3037] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:32.295878 19162 raft_consensus.cc:504] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.296543 19162 leader_election.cc:288] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [CANDIDATE]: Term 1 election: Requested vote from peers 66a009cf596b46c8b1a7e5d0325193c7 (127.0.97.2:43573), 382a454661834f3e9ff6e0e114b7b10e (127.0.97.1:44309)
I20260923 21:05:32.297030 18957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9929618d3d4726a54cde382cfc3867" candidate_uuid: "b2a3a6e339e84798ae6a93319cacb649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a009cf596b46c8b1a7e5d0325193c7"
I20260923 21:05:32.297106 18823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d9929618d3d4726a54cde382cfc3867" candidate_uuid: "b2a3a6e339e84798ae6a93319cacb649" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382a454661834f3e9ff6e0e114b7b10e"
I20260923 21:05:32.297271 18957 raft_consensus.cc:3037] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:32.297366 18823 raft_consensus.cc:3037] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:32.299613 18957 raft_consensus.cc:2454] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a3a6e339e84798ae6a93319cacb649 in term 1.
I20260923 21:05:32.299626 18823 raft_consensus.cc:2454] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2a3a6e339e84798ae6a93319cacb649 in term 1.
I20260923 21:05:32.300082 19026 leader_election.cc:302] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 382a454661834f3e9ff6e0e114b7b10e, b2a3a6e339e84798ae6a93319cacb649; no voters:
I20260923 21:05:32.300416 19162 raft_consensus.cc:2781] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:32.300894 19162 raft_consensus.cc:686] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 1 LEADER]: Becoming Leader. State: Replica: b2a3a6e339e84798ae6a93319cacb649, State: Running, Role: LEADER
I20260923 21:05:32.301456 19162 consensus_queue.cc:237] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } }
I20260923 21:05:32.306143 18679 catalog_manager.cc:5495] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a3a6e339e84798ae6a93319cacb649 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b2a3a6e339e84798ae6a93319cacb649" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2a3a6e339e84798ae6a93319cacb649" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 } health_report { overall_health: UNKNOWN } } }
I20260923 21:05:32.341400 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.336058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.0.97.3:46447)
0923 21:05:32.336241 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.336246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.336264 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:05:32.336354 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.336357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.336433 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.336569 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:05:32.336983 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.337745 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.339202 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.339600 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.339705 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.340197 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:05:32.340741 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:05:32.340919 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.341085 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.341129 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 21:05:32.350735 18823 raft_consensus.cc:1261] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 1 FOLLOWER]: Refusing update from remote peer b2a3a6e339e84798ae6a93319cacb649: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:32.350734 18957 raft_consensus.cc:1261] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 1 FOLLOWER]: Refusing update from remote peer b2a3a6e339e84798ae6a93319cacb649: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:32.351686 19162 consensus_queue.cc:1035] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a009cf596b46c8b1a7e5d0325193c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:32.351992 19165 consensus_queue.cc:1035] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [LEADER]: Connected to new peer: Peer: permanent_uuid: "382a454661834f3e9ff6e0e114b7b10e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:32.372537 19170 mvcc.cc:205] Tried to move back new op lower bound from 7332649092497555456 to 7332649092311171072. Current Snapshot: MvccSnapshot[applied={T|T < 7332649092497555456}]
I20260923 21:05:32.389331 18678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:32.389714 18678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:32.393981 18678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f50f7db78934090abaa9b51238347d6: Sending DeleteTablet for 3 replicas of tablet 4d9929618d3d4726a54cde382cfc3867
I20260923 21:05:32.395421 18937 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9929618d3d4726a54cde382cfc3867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40306
I20260923 21:05:32.395457 18803 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9929618d3d4726a54cde382cfc3867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45940
I20260923 21:05:32.396309 19071 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d9929618d3d4726a54cde382cfc3867 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46022
I20260923 21:05:32.398689 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 18740
I20260923 21:05:32.403570 19179 tablet_replica.cc:330] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: stopping tablet replica
I20260923 21:05:32.403853 19180 tablet_replica.cc:330] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: stopping tablet replica
I20260923 21:05:32.404081 19179 raft_consensus.cc:2229] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:32.404495 19180 raft_consensus.cc:2229] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:05:32.404704 19179 raft_consensus.cc:2258] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:32.405439 19180 raft_consensus.cc:2258] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:32.405645 19178 tablet_replica.cc:330] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: stopping tablet replica
I20260923 21:05:32.406137 19178 raft_consensus.cc:2229] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:32.406750 19178 raft_consensus.cc:2258] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:32.408003 19179 ts_tablet_manager.cc:1905] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:32.408109 19180 ts_tablet_manager.cc:1905] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:32.416666 19180 ts_tablet_manager.cc:1918] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:32.417013 19180 log.cc:1196] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/4d9929618d3d4726a54cde382cfc3867
I20260923 21:05:32.417722 19180 ts_tablet_manager.cc:1939] T 4d9929618d3d4726a54cde382cfc3867 P b2a3a6e339e84798ae6a93319cacb649: Deleting consensus metadata
I20260923 21:05:32.419195 19179 ts_tablet_manager.cc:1918] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:32.419406 19179 log.cc:1196] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/4d9929618d3d4726a54cde382cfc3867
I20260923 21:05:32.419770 19178 ts_tablet_manager.cc:1905] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:32.420017 19179 ts_tablet_manager.cc:1939] T 4d9929618d3d4726a54cde382cfc3867 P 66a009cf596b46c8b1a7e5d0325193c7: Deleting consensus metadata
I20260923 21:05:32.420413 18665 catalog_manager.cc:4841] TS b2a3a6e339e84798ae6a93319cacb649 (127.0.97.3:46447): tablet 4d9929618d3d4726a54cde382cfc3867 (table test-table [id=b8ce580f98324309842ce495c36bd8dc]) successfully deleted
I20260923 21:05:32.421731 18664 catalog_manager.cc:4841] TS 66a009cf596b46c8b1a7e5d0325193c7 (127.0.97.2:43573): tablet 4d9929618d3d4726a54cde382cfc3867 (table test-table [id=b8ce580f98324309842ce495c36bd8dc]) successfully deleted
I20260923 21:05:32.426306 19178 ts_tablet_manager.cc:1918] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:32.426589 19178 log.cc:1196] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/4d9929618d3d4726a54cde382cfc3867
I20260923 21:05:32.427201 19178 ts_tablet_manager.cc:1939] T 4d9929618d3d4726a54cde382cfc3867 P 382a454661834f3e9ff6e0e114b7b10e: Deleting consensus metadata
I20260923 21:05:32.429375 18665 catalog_manager.cc:4841] TS 382a454661834f3e9ff6e0e114b7b10e (127.0.97.1:44309): tablet 4d9929618d3d4726a54cde382cfc3867 (table test-table [id=b8ce580f98324309842ce495c36bd8dc]) successfully deleted
Sep 23 21:05:32 dist-test-slave-8xn5 krb5kdc[18634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:05:32.437234 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.403253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44950 (local address 127.0.97.1:44309)
0923 21:05:32.403509 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.403519 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.419054 (+ 15535us) server_negotiation.cc:407] Connection header received
0923 21:05:32.419427 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.419436 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.419542 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.419629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:32.421519 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.422856 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.424811 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.425605 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.425766 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.431173 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:32.431186 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:32.431200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:32.431237 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:32.434204 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.435038 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.435049 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.435057 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.435195 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.435585 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.435590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.435593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.435966 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:32.436286 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.436592 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.436823 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 21:05:32.438017 18863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:32.529918 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 18740
I20260923 21:05:32.557999 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 18874
Sep 23 21:05:32 dist-test-slave-8xn5 krb5kdc[18634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:05:32.577369 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.559027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.0.97.2:43573)
0923 21:05:32.559198 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.559202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.559220 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:05:32.559356 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.559360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.559450 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.559539 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:32.561355 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.562088 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.564267 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.564653 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.564748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.570626 (+ 5878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:32.570643 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:32.570660 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:32.570699 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:32.574187 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.575136 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.575147 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.575151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.575309 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.575710 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.575716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.575719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.576146 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:32.576465 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.576763 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.576992 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 21:05:32.578073 18997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:32.682273 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 18874
I20260923 21:05:32.706835 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19008
Sep 23 21:05:32 dist-test-slave-8xn5 krb5kdc[18634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:05:32.724295 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:32.707874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.97.3:46447)
0923 21:05:32.708082 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:32.708088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:32.708111 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:05:32.708221 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:32.708231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:32.708308 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:32.708450 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:32.709922 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.710862 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.712246 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:32.712738 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:32.712948 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:32.717895 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:32.717909 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:32.717925 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:32.717969 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:32.721216 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.721931 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.721939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.721941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.722090 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:32.722589 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:32.722594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:32.722604 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:32.722979 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:32.723336 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:32.723655 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 21:05:32.723903 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 21:05:32.725208 19131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:32.849098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19008
I20260923 21:05:32.868834 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 18647
I20260923 21:05:32.870105 18709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:32.983672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 18647
2026-09-23T21:05:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4586 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T21:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:33Z Disabled control of system clock
I20260923 21:05:33.063894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40303
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 21:05:33.429080 19189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:33.429405 19189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:33.429488 19189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:33.437515 19189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:33.437677 19189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:33.437724 19189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:33.437776 19189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:33.437824 19189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:33.447778 19189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:33.449184 19189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:33.451068 19189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:33.463786 19194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:33.463811 19197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:33.463757 19195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:33.465008 19189 server_base.cc:1034] running on GCE node
I20260923 21:05:33.465948 19189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:33.467561 19189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:33.468779 19189 hybrid_clock.cc:648] HybridClock initialized: now 1790197533468742 us; error 33 us; skew 500 ppm
I20260923 21:05:33.469264 19189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:33.471915 19189 webserver.cc:460] Webserver started at http://127.0.97.62:33489/ using document root <none> and password file <none>
I20260923 21:05:33.472668 19189 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:33.472761 19189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:33.473117 19189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:33.476363 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "9ef9ccaf8e104cecafc30b465011a92e"
format_stamp: "Formatted at 2026-09-23 21:05:33 on dist-test-slave-8xn5"
server_key: "ee3372799b220543972825c625e0e332"
server_key_iv: "0c5e9af6e34934f078d0a226768f15c0"
server_key_version: "encryptionkey@0"
I20260923 21:05:33.477255 19189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "9ef9ccaf8e104cecafc30b465011a92e"
format_stamp: "Formatted at 2026-09-23 21:05:33 on dist-test-slave-8xn5"
server_key: "ee3372799b220543972825c625e0e332"
server_key_iv: "0c5e9af6e34934f078d0a226768f15c0"
server_key_version: "encryptionkey@0"
I20260923 21:05:33.485232 19189 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260923 21:05:33.490149 19203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:33.491947 19189 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:05:33.492183 19189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "9ef9ccaf8e104cecafc30b465011a92e"
format_stamp: "Formatted at 2026-09-23 21:05:33 on dist-test-slave-8xn5"
server_key: "ee3372799b220543972825c625e0e332"
server_key_iv: "0c5e9af6e34934f078d0a226768f15c0"
server_key_version: "encryptionkey@0"
I20260923 21:05:33.492447 19189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:33.517060 19189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:33.517988 19189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:33.518299 19189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:33.538225 19189 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40303
I20260923 21:05:33.538244 19254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40303 every 8 connection(s)
I20260923 21:05:33.539710 19189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:33.540884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19189
I20260923 21:05:33.541072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:33.541617 388 external_mini_cluster.cc:1428] Setting key c4195853b1082f69bd020fec0fcac918
I20260923 21:05:33.545279 19255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:33.555514 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:33.543783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.0.97.62:40303)
0923 21:05:33.544788 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:33.544799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:33.544852 (+ 53us) server_negotiation.cc:407] Connection header received
0923 21:05:33.546699 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:33.546722 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:33.547182 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:33.547590 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:33.548179 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:33.549481 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:33.550948 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:33.551632 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:33.551839 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:33.553080 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:33.553089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:33.553103 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:33.553682 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:33.553759 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:33.553787 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:33.553944 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:33.554185 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 21:05:33.554560 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":355,"threads_started":1}
I20260923 21:05:33.556567 19255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e: Bootstrap starting.
I20260923 21:05:33.559895 19255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:33.561002 19255 log.cc:824] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:33.563936 19255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e: No bootstrap required, opened a new log
I20260923 21:05:33.571484 19255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef9ccaf8e104cecafc30b465011a92e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260923 21:05:33.571946 19255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:33.572062 19255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef9ccaf8e104cecafc30b465011a92e, State: Initialized, Role: FOLLOWER
I20260923 21:05:33.572683 19255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef9ccaf8e104cecafc30b465011a92e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260923 21:05:33.572958 19255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:33.573108 19255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:33.573289 19255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:33.575733 19255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef9ccaf8e104cecafc30b465011a92e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260923 21:05:33.576292 19255 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef9ccaf8e104cecafc30b465011a92e; no voters:
I20260923 21:05:33.576949 19255 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:33.577162 19260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:33.578198 19260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [term 1 LEADER]: Becoming Leader. State: Replica: 9ef9ccaf8e104cecafc30b465011a92e, State: Running, Role: LEADER
I20260923 21:05:33.578825 19260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef9ccaf8e104cecafc30b465011a92e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } }
I20260923 21:05:33.579164 19255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:33.581805 19262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef9ccaf8e104cecafc30b465011a92e. Latest consensus state: current_term: 1 leader_uuid: "9ef9ccaf8e104cecafc30b465011a92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef9ccaf8e104cecafc30b465011a92e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } } }
I20260923 21:05:33.582227 19262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:33.582577 19261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef9ccaf8e104cecafc30b465011a92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef9ccaf8e104cecafc30b465011a92e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40303 } } }
I20260923 21:05:33.582888 19261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:33.586302 19269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:33.590986 19269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:33.600205 19269 catalog_manager.cc:1347] Generated new cluster ID: 2e5fd7f2a350488589c6d61fbd369a93
I20260923 21:05:33.600342 19269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:33.614140 19269 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:33.615574 19269 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:33.625250 19269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e: Generated new TSK 0
I20260923 21:05:33.626514 19269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:33.643823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 21:05:34.004124 19279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:34.004472 19279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:34.004648 19279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:34.012691 19279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:34.012835 19279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:34.013036 19279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:05:34.023300 19279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:34.024837 19279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:34.027009 19279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:34.042913 19287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:34.043385 19284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:34.043901 19279 server_base.cc:1034] running on GCE node
W20260923 21:05:34.043962 19285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:34.044728 19279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:34.045750 19279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:34.046975 19279 hybrid_clock.cc:648] HybridClock initialized: now 1790197534046926 us; error 43 us; skew 500 ppm
I20260923 21:05:34.047354 19279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:34.049765 19279 webserver.cc:460] Webserver started at http://127.0.97.1:40277/ using document root <none> and password file <none>
I20260923 21:05:34.050686 19279 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:34.050813 19279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:34.051177 19279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:34.054261 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "af946a6fc6da4f6982dbc3fba6e79ad5"
format_stamp: "Formatted at 2026-09-23 21:05:34 on dist-test-slave-8xn5"
server_key: "3ecadf90dfbc5e40a9216349d1f20ff7"
server_key_iv: "21ead35723d314a94ba76cd78afe2190"
server_key_version: "encryptionkey@0"
I20260923 21:05:34.055295 19279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "af946a6fc6da4f6982dbc3fba6e79ad5"
format_stamp: "Formatted at 2026-09-23 21:05:34 on dist-test-slave-8xn5"
server_key: "3ecadf90dfbc5e40a9216349d1f20ff7"
server_key_iv: "21ead35723d314a94ba76cd78afe2190"
server_key_version: "encryptionkey@0"
I20260923 21:05:34.060711 19279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 21:05:34.064603 19293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:34.066227 19279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:05:34.066535 19279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "af946a6fc6da4f6982dbc3fba6e79ad5"
format_stamp: "Formatted at 2026-09-23 21:05:34 on dist-test-slave-8xn5"
server_key: "3ecadf90dfbc5e40a9216349d1f20ff7"
server_key_iv: "21ead35723d314a94ba76cd78afe2190"
server_key_version: "encryptionkey@0"
I20260923 21:05:34.066752 19279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:34.090746 19279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:34.091645 19279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:34.091943 19279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:34.093096 19279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:34.094846 19279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:34.094938 19279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:34.095095 19279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:34.095191 19279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:34.130635 19279 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35597
I20260923 21:05:34.130738 19405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35597 every 8 connection(s)
I20260923 21:05:34.131973 19279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:05:34.133819 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19279
I20260923 21:05:34.134004 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:05:34.134500 388 external_mini_cluster.cc:1428] Setting key 14e0f5baf596746a830b4963fbd825dd
I20260923 21:05:34.138563 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 21:05:34.147948 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:34.134786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59659 (local address 127.0.97.62:40303)
0923 21:05:34.135287 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:34.135292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:34.136690 (+ 1398us) server_negotiation.cc:407] Connection header received
0923 21:05:34.139389 (+ 2699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:34.139394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:34.139474 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:34.139567 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:34.141196 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.141998 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.144571 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.144991 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.145093 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:34.146950 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:34.146956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:34.146959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:34.146980 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:34.147021 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:34.147059 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:34.147166 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:34.147409 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 21:05:34.147532 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
I20260923 21:05:34.148784 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:34.135447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40303 (local address 127.0.97.1:59659)
0923 21:05:34.136499 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:34.136551 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:34.139062 (+ 2511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:34.139933 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:34.139949 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:34.140108 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:34.141000 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:34.141031 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.142185 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.142193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:34.144409 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:34.144428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.145234 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.145244 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:34.145604 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:34.146540 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:34.146546 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:34.146603 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:34.146722 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:34.147181 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:34.147190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:34.147196 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 21:05:34.147343 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 21:05:34.147658 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":438,"threads_started":1}
I20260923 21:05:34.150553 19406 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40303
I20260923 21:05:34.150895 19406 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:34.151835 19406 heartbeater.cc:511] Master 127.0.97.62:40303 requested a full tablet report, sending...
I20260923 21:05:34.154196 19220 ts_manager.cc:194] Registered new tserver with Master: af946a6fc6da4f6982dbc3fba6e79ad5 (127.0.97.1:35597)
I20260923 21:05:34.155809 19220 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59659
W20260923 21:05:34.492859 19410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:34.493249 19410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:34.493419 19410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:34.501302 19410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:34.501452 19410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:34.501638 19410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:05:34.511703 19410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:34.513195 19410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:34.515089 19410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:34.530004 19416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:34.530005 19415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:34.530226 19418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:34.530853 19410 server_base.cc:1034] running on GCE node
I20260923 21:05:34.531554 19410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:34.532696 19410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:34.533986 19410 hybrid_clock.cc:648] HybridClock initialized: now 1790197534533931 us; error 56 us; skew 500 ppm
I20260923 21:05:34.534528 19410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:34.537523 19410 webserver.cc:460] Webserver started at http://127.0.97.2:44547/ using document root <none> and password file <none>
I20260923 21:05:34.538563 19410 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:34.538722 19410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:34.539100 19410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:34.543351 19410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "8cd97855a99f48948221819977820a9a"
format_stamp: "Formatted at 2026-09-23 21:05:34 on dist-test-slave-8xn5"
server_key: "7ee129c8bdf4b722f9b1a56390498d01"
server_key_iv: "79d2b88542185e8eff94be0cdd840a1a"
server_key_version: "encryptionkey@0"
I20260923 21:05:34.544461 19410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8cd97855a99f48948221819977820a9a"
format_stamp: "Formatted at 2026-09-23 21:05:34 on dist-test-slave-8xn5"
server_key: "7ee129c8bdf4b722f9b1a56390498d01"
server_key_iv: "79d2b88542185e8eff94be0cdd840a1a"
server_key_version: "encryptionkey@0"
I20260923 21:05:34.551661 19410 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260923 21:05:34.556680 19424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:34.558503 19410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:05:34.558701 19410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "8cd97855a99f48948221819977820a9a"
format_stamp: "Formatted at 2026-09-23 21:05:34 on dist-test-slave-8xn5"
server_key: "7ee129c8bdf4b722f9b1a56390498d01"
server_key_iv: "79d2b88542185e8eff94be0cdd840a1a"
server_key_version: "encryptionkey@0"
I20260923 21:05:34.558974 19410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:34.580077 19410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:34.581036 19410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:34.581300 19410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:34.582401 19410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:34.584064 19410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:34.584159 19410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:34.584239 19410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:34.584295 19410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:34.619639 19410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36691
I20260923 21:05:34.619642 19536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36691 every 8 connection(s)
I20260923 21:05:34.621030 19410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:05:34.631036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19410
I20260923 21:05:34.631305 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:05:34.631770 388 external_mini_cluster.cc:1428] Setting key 54cb03e297de9d08d39b8f49ba63a72b
I20260923 21:05:34.635143 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:34.623972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47671 (local address 127.0.97.62:40303)
0923 21:05:34.624229 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:34.624236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:34.626029 (+ 1793us) server_negotiation.cc:407] Connection header received
0923 21:05:34.627945 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:34.627950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:34.628025 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:34.628128 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:34.629272 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.630025 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.631893 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.632488 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.632647 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:34.634162 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:34.634169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:34.634173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:34.634195 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:34.634244 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:34.634262 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:34.634386 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:34.634647 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 21:05:34.634758 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 21:05:34.635231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--builtin_ntp_servers=127.0.97.20:45439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 21:05:34.636242 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:34.624692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40303 (local address 127.0.97.2:47671)
0923 21:05:34.625847 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:34.625887 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:34.627645 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:34.628396 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:34.628405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:34.628502 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:34.629076 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:34.629092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.630180 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.630184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:34.631727 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:34.631741 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:34.632739 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:34.632744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:34.632978 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:34.633835 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:34.633838 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:34.633882 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:34.633973 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:34.634417 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:34.634426 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:34.634432 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 21:05:34.634591 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 21:05:34.634944 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":341,"threads_started":1}
I20260923 21:05:34.637833 19537 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40303
I20260923 21:05:34.638180 19537 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:34.639204 19537 heartbeater.cc:511] Master 127.0.97.62:40303 requested a full tablet report, sending...
I20260923 21:05:34.640964 19220 ts_manager.cc:194] Registered new tserver with Master: 8cd97855a99f48948221819977820a9a (127.0.97.2:36691)
I20260923 21:05:34.642189 19220 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:47671
W20260923 21:05:34.980156 19540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:34.980476 19540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:34.980610 19540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:34.988410 19540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:34.988528 19540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:34.988711 19540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:05:34.998651 19540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:35.000037 19540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:35.001891 19540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:35.016367 19546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:35.016531 19545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:35.016566 19548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:35.018196 19540 server_base.cc:1034] running on GCE node
I20260923 21:05:35.018956 19540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:35.020211 19540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:35.021507 19540 hybrid_clock.cc:648] HybridClock initialized: now 1790197535021457 us; error 43 us; skew 500 ppm
I20260923 21:05:35.021905 19540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:35.024607 19540 webserver.cc:460] Webserver started at http://127.0.97.3:36921/ using document root <none> and password file <none>
I20260923 21:05:35.025393 19540 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:35.025532 19540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:35.025885 19540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:35.029103 19540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "0df0802600e0440ab75282455de957e0"
format_stamp: "Formatted at 2026-09-23 21:05:35 on dist-test-slave-8xn5"
server_key: "3938e4ddfdf0064252372863fb425641"
server_key_iv: "d37e962b08c6357d4a215f3b41a1dc94"
server_key_version: "encryptionkey@0"
I20260923 21:05:35.029961 19540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0df0802600e0440ab75282455de957e0"
format_stamp: "Formatted at 2026-09-23 21:05:35 on dist-test-slave-8xn5"
server_key: "3938e4ddfdf0064252372863fb425641"
server_key_iv: "d37e962b08c6357d4a215f3b41a1dc94"
server_key_version: "encryptionkey@0"
I20260923 21:05:35.035847 19540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 21:05:35.040241 19554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:35.042003 19540 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 21:05:35.042234 19540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "0df0802600e0440ab75282455de957e0"
format_stamp: "Formatted at 2026-09-23 21:05:35 on dist-test-slave-8xn5"
server_key: "3938e4ddfdf0064252372863fb425641"
server_key_iv: "d37e962b08c6357d4a215f3b41a1dc94"
server_key_version: "encryptionkey@0"
I20260923 21:05:35.042554 19540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:35.069515 19540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:35.070489 19540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:35.070778 19540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:35.071923 19540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:35.073532 19540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:35.073622 19540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:35.073786 19540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:35.073866 19540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:35.108072 19540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33503
I20260923 21:05:35.108139 19666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33503 every 8 connection(s)
I20260923 21:05:35.109601 19540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:05:35.113636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19540
I20260923 21:05:35.113902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:05:35.114475 388 external_mini_cluster.cc:1428] Setting key 1312cef7d7da2c68781d0249d1687c6b
I20260923 21:05:35.123718 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.112506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37173 (local address 127.0.97.62:40303)
0923 21:05:35.112729 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.112734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.114612 (+ 1878us) server_negotiation.cc:407] Connection header received
0923 21:05:35.116778 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.116784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.116885 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.117003 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:35.118396 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.119142 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.120754 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.121168 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.121320 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.122818 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:35.122826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:35.122831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:35.122855 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:35.122897 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:35.122913 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:35.123025 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.123257 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.123374 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 21:05:35.124835 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.113102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40303 (local address 127.0.97.3:37173)
0923 21:05:35.114377 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.114424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.116496 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.117261 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.117270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.117383 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:35.118162 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.118181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.119305 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.119311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.120600 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.120612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.121411 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.121419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.121665 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.122447 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:35.122449 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:35.122499 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:35.122587 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:35.123025 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:35.123038 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:35.123049 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.123246 (+ 197us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.123710 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":440,"threads_started":1}
I20260923 21:05:35.126387 19667 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40303
I20260923 21:05:35.126675 19667 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:35.127414 19667 heartbeater.cc:511] Master 127.0.97.62:40303 requested a full tablet report, sending...
I20260923 21:05:35.128921 19220 ts_manager.cc:194] Registered new tserver with Master: 0df0802600e0440ab75282455de957e0 (127.0.97.3:33503)
I20260923 21:05:35.129930 19220 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:37173
I20260923 21:05:35.130451 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:05:35.142397 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.135875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.0.97.62:40303)
0923 21:05:35.136115 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.136125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.136577 (+ 452us) server_negotiation.cc:407] Connection header received
0923 21:05:35.136829 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.136835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.136924 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.137044 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:35.137657 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.138427 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.139875 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.140265 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.140365 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.141518 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:35.141523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:35.141526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:35.141545 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:35.141578 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:35.141589 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:35.141701 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.141924 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.142003 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:05:35.146891 19220 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:05:35.153777 19220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:05:35.159809 19406 heartbeater.cc:503] Master 127.0.97.62:40303 was elected leader, sending a full tablet report...
I20260923 21:05:35.181177 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.169218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54402)
0923 21:05:35.172948 (+ 3730us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.172972 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.173106 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.174279 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.174284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.174312 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:35.174712 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.174724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.176572 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.176577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.178148 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.178163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.180093 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.180099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.180332 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.180392 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.180575 (+ 183us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.180781 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3554,"mutex_wait_us":1,"thread_start_us":3372,"threads_started":1}
I20260923 21:05:35.181979 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.168028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40608 (local address 127.0.97.2:36691)
0923 21:05:35.169489 (+ 1461us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.169503 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.169543 (+ 40us) server_negotiation.cc:407] Connection header received
0923 21:05:35.169670 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.169683 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.170114 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.170382 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:35.173689 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.175115 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.178773 (+ 3658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.180490 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.180626 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.180882 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.181450 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.181633 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":1488,"threads_started":1}
I20260923 21:05:35.182077 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.169787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.0.97.1:35597)
0923 21:05:35.173666 (+ 3879us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.173683 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.173722 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:05:35.173825 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.173833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.174063 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.174274 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:35.174877 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.176435 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.178385 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.179882 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.180023 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.180334 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.181518 (+ 1184us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.181707 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3695,"thread_start_us":3550,"threads_started":1}
I20260923 21:05:35.183429 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.167528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.0.97.3:33503)
0923 21:05:35.171492 (+ 3964us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.171501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.171537 (+ 36us) server_negotiation.cc:407] Connection header received
0923 21:05:35.171637 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.171646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.171985 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.172179 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:35.173326 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.175249 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.180457 (+ 5208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.181919 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.182075 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.182377 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.182926 (+ 549us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.183093 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3780,"thread_start_us":3333,"threads_started":1}
I20260923 21:05:35.182914 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.167485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36691 (local address 127.0.0.1:40608)
0923 21:05:35.169137 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.169152 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.169249 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.172932 (+ 3683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.172941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.172984 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:35.173494 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.173505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.175301 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.175307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.178524 (+ 3217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.178540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.181024 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.181031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.181238 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.181271 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.182430 (+ 1159us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.182576 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1491,"mutex_wait_us":402,"thread_start_us":353,"threads_started":1}
I20260923 21:05:35.183900 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.167326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33503 (local address 127.0.0.1:59240)
0923 21:05:35.168260 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.168335 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.168539 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.172453 (+ 3914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.172463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.172508 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:35.173093 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.173106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.175426 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.175438 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.178524 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.178540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.182131 (+ 3591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.182138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.182358 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.182391 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.182484 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.182639 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":281,"thread_start_us":1508,"threads_started":1}
I20260923 21:05:35.186513 19341 tablet_service.cc:1467] Processing CreateTablet for tablet b3ea324b0472493bb082908a014bd581 (DEFAULT_TABLE table=test-table [id=da8971ce683445299ae95584e3cea55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:35.187858 19341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ea324b0472493bb082908a014bd581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:35.187546 19602 tablet_service.cc:1467] Processing CreateTablet for tablet b3ea324b0472493bb082908a014bd581 (DEFAULT_TABLE table=test-table [id=da8971ce683445299ae95584e3cea55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:35.187546 19472 tablet_service.cc:1467] Processing CreateTablet for tablet b3ea324b0472493bb082908a014bd581 (DEFAULT_TABLE table=test-table [id=da8971ce683445299ae95584e3cea55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:35.188982 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ea324b0472493bb082908a014bd581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:35.188985 19472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3ea324b0472493bb082908a014bd581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:35.201551 19685 tablet_bootstrap.cc:492] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: Bootstrap starting.
I20260923 21:05:35.204324 19686 tablet_bootstrap.cc:492] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Bootstrap starting.
I20260923 21:05:35.205210 19687 tablet_bootstrap.cc:492] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Bootstrap starting.
I20260923 21:05:35.205675 19685 tablet_bootstrap.cc:654] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:35.207194 19685 log.cc:824] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:35.209254 19687 tablet_bootstrap.cc:654] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:35.210604 19686 tablet_bootstrap.cc:654] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:35.210739 19687 log.cc:824] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:35.211094 19685 tablet_bootstrap.cc:492] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: No bootstrap required, opened a new log
I20260923 21:05:35.211468 19685 ts_tablet_manager.cc:1397] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 21:05:35.211915 19686 log.cc:824] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:35.215822 19686 tablet_bootstrap.cc:492] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: No bootstrap required, opened a new log
I20260923 21:05:35.216143 19686 ts_tablet_manager.cc:1397] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260923 21:05:35.216805 19687 tablet_bootstrap.cc:492] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: No bootstrap required, opened a new log
I20260923 21:05:35.217087 19687 ts_tablet_manager.cc:1397] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260923 21:05:35.222318 19685 raft_consensus.cc:348] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.222900 19685 raft_consensus.cc:374] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:35.223071 19685 raft_consensus.cc:729] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af946a6fc6da4f6982dbc3fba6e79ad5, State: Initialized, Role: FOLLOWER
I20260923 21:05:35.223796 19685 consensus_queue.cc:260] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.225842 19685 ts_tablet_manager.cc:1428] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260923 21:05:35.226408 19687 raft_consensus.cc:348] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.226972 19687 raft_consensus.cc:374] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:35.227154 19687 raft_consensus.cc:729] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd97855a99f48948221819977820a9a, State: Initialized, Role: FOLLOWER
I20260923 21:05:35.226953 19686 raft_consensus.cc:348] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.227429 19686 raft_consensus.cc:374] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:35.227552 19686 raft_consensus.cc:729] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df0802600e0440ab75282455de957e0, State: Initialized, Role: FOLLOWER
I20260923 21:05:35.227929 19687 consensus_queue.cc:260] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.228147 19686 consensus_queue.cc:260] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.229221 19537 heartbeater.cc:503] Master 127.0.97.62:40303 was elected leader, sending a full tablet report...
I20260923 21:05:35.230060 19687 ts_tablet_manager.cc:1428] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260923 21:05:35.231552 19667 heartbeater.cc:503] Master 127.0.97.62:40303 was elected leader, sending a full tablet report...
I20260923 21:05:35.232167 19686 ts_tablet_manager.cc:1428] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260923 21:05:35.326930 19691 raft_consensus.cc:482] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:35.327320 19691 raft_consensus.cc:504] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.329159 19691 leader_election.cc:288] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0df0802600e0440ab75282455de957e0 (127.0.97.3:33503), 8cd97855a99f48948221819977820a9a (127.0.97.2:36691)
I20260923 21:05:35.337370 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.329603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33503 (local address 127.0.97.1:50131)
0923 21:05:35.330740 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.330756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.330886 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.331266 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.331271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.331304 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:35.331817 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.331834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.333042 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.333048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.335100 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.335116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.336418 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.336425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.336606 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.336684 (+ 78us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.336768 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.336922 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":23,"thread_start_us":815,"threads_started":1}
I20260923 21:05:35.337252 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.329614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50131 (local address 127.0.97.3:33503)
0923 21:05:35.329846 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.329853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.330858 (+ 1005us) server_negotiation.cc:407] Connection header received
0923 21:05:35.331025 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.331030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.331116 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.331207 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:35.331996 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.332895 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.335329 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.336198 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.336308 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.336421 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.336828 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.336966 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:05:35.337424 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.329644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49205 (local address 127.0.97.2:36691)
0923 21:05:35.329857 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.329864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.330987 (+ 1123us) server_negotiation.cc:407] Connection header received
0923 21:05:35.331263 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.331269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.331389 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.331517 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:35.332268 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.333294 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.335464 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.336312 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.336456 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.336552 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.336908 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.337042 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 21:05:35.337419 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.329603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36691 (local address 127.0.97.1:49205)
0923 21:05:35.330858 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.330878 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.331029 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.331565 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.331568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.331594 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:35.331984 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.331995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.333516 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.333525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.335107 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.335132 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.336523 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.336528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.336716 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.336762 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.336861 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.336986 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":135,"thread_start_us":527,"threads_started":1}
I20260923 21:05:35.338065 19692 raft_consensus.cc:482] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:35.338462 19692 raft_consensus.cc:504] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.338793 19622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ea324b0472493bb082908a014bd581" candidate_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df0802600e0440ab75282455de957e0" is_pre_election: true
I20260923 21:05:35.339212 19492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ea324b0472493bb082908a014bd581" candidate_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd97855a99f48948221819977820a9a" is_pre_election: true
I20260923 21:05:35.339378 19622 raft_consensus.cc:2454] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af946a6fc6da4f6982dbc3fba6e79ad5 in term 0.
I20260923 21:05:35.339676 19492 raft_consensus.cc:2454] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af946a6fc6da4f6982dbc3fba6e79ad5 in term 0.
I20260923 21:05:35.340206 19296 leader_election.cc:302] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df0802600e0440ab75282455de957e0, af946a6fc6da4f6982dbc3fba6e79ad5; no voters:
I20260923 21:05:35.340631 19692 leader_election.cc:288] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0df0802600e0440ab75282455de957e0 (127.0.97.3:33503), af946a6fc6da4f6982dbc3fba6e79ad5 (127.0.97.1:35597)
I20260923 21:05:35.340659 19691 raft_consensus.cc:2781] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:05:35.340816 19691 raft_consensus.cc:482] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:05:35.340965 19691 raft_consensus.cc:3037] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:35.344574 19691 raft_consensus.cc:504] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.345597 19691 leader_election.cc:288] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [CANDIDATE]: Term 1 election: Requested vote from peers 0df0802600e0440ab75282455de957e0 (127.0.97.3:33503), 8cd97855a99f48948221819977820a9a (127.0.97.2:36691)
I20260923 21:05:35.346159 19492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ea324b0472493bb082908a014bd581" candidate_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd97855a99f48948221819977820a9a"
I20260923 21:05:35.346154 19622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ea324b0472493bb082908a014bd581" candidate_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df0802600e0440ab75282455de957e0"
I20260923 21:05:35.346431 19492 raft_consensus.cc:3037] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:35.346436 19622 raft_consensus.cc:3037] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:35.349247 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.341277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36607 (local address 127.0.97.3:33503)
0923 21:05:35.341479 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.341485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.341585 (+ 100us) server_negotiation.cc:407] Connection header received
0923 21:05:35.341884 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.341890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.341972 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.342033 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:35.342816 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.343973 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.346565 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.347937 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.348047 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.348165 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.348802 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.348960 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 21:05:35.349423 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.341773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42923 (local address 127.0.97.1:35597)
0923 21:05:35.341932 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.341937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.342391 (+ 454us) server_negotiation.cc:407] Connection header received
0923 21:05:35.342659 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.342664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.342777 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.342875 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:35.343392 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.344135 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.346956 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.348205 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.348324 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.348419 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.349033 (+ 614us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.349174 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 21:05:35.350015 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.340859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33503 (local address 127.0.97.2:36607)
0923 21:05:35.341493 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.341514 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.341695 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.342226 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.342229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.342251 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:35.342662 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.342677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.344129 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.344135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.346298 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.346314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.348396 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.348402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.348589 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.348630 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.349465 (+ 835us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.349610 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":52,"spinlock_wait_cycles":30464,"thread_start_us":290,"threads_started":1}
I20260923 21:05:35.350207 19697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.341644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.97.2:42923)
0923 21:05:35.342240 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.342268 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:35.342488 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:35.342887 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:35.342891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:35.342913 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:35.343244 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.343252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.344305 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.344313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.346714 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:35.346729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.348641 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.348649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:35.348833 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.348882 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 21:05:35.349590 (+ 708us) client_negotiation.cc:240] Negotiation successful
0923 21:05:35.349754 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":6,"spinlock_wait_cycles":4352,"thread_start_us":299,"threads_started":1}
I20260923 21:05:35.350418 19622 raft_consensus.cc:2454] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af946a6fc6da4f6982dbc3fba6e79ad5 in term 1.
I20260923 21:05:35.350876 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ea324b0472493bb082908a014bd581" candidate_uuid: "8cd97855a99f48948221819977820a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df0802600e0440ab75282455de957e0" is_pre_election: true
I20260923 21:05:35.351078 19361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3ea324b0472493bb082908a014bd581" candidate_uuid: "8cd97855a99f48948221819977820a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" is_pre_election: true
I20260923 21:05:35.351231 19621 raft_consensus.cc:2379] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8cd97855a99f48948221819977820a9a in current term 1: Already voted for candidate af946a6fc6da4f6982dbc3fba6e79ad5 in this term.
I20260923 21:05:35.351581 19361 raft_consensus.cc:2379] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8cd97855a99f48948221819977820a9a in current term 1: Already voted for candidate af946a6fc6da4f6982dbc3fba6e79ad5 in this term.
I20260923 21:05:35.350139 19492 raft_consensus.cc:2454] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af946a6fc6da4f6982dbc3fba6e79ad5 in term 1.
I20260923 21:05:35.352180 19296 leader_election.cc:302] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df0802600e0440ab75282455de957e0, af946a6fc6da4f6982dbc3fba6e79ad5; no voters:
I20260923 21:05:35.352517 19691 raft_consensus.cc:2781] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:35.352588 19427 leader_election.cc:302] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8cd97855a99f48948221819977820a9a; no voters: 0df0802600e0440ab75282455de957e0, af946a6fc6da4f6982dbc3fba6e79ad5
I20260923 21:05:35.353153 19692 raft_consensus.cc:2726] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 21:05:35.353139 19691 raft_consensus.cc:686] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 1 LEADER]: Becoming Leader. State: Replica: af946a6fc6da4f6982dbc3fba6e79ad5, State: Running, Role: LEADER
I20260923 21:05:35.353670 19691 consensus_queue.cc:237] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } }
I20260923 21:05:35.358721 19219 catalog_manager.cc:5495] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 reported cstate change: term changed from 0 to 1, leader changed from <none> to af946a6fc6da4f6982dbc3fba6e79ad5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af946a6fc6da4f6982dbc3fba6e79ad5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35597 } health_report { overall_health: HEALTHY } } }
W20260923 21:05:35.361455 19668 tablet.cc:2369] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:35.373304 19538 tablet.cc:2369] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:35.378650 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.372390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.0.97.1:35597)
0923 21:05:35.372583 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.372588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.372905 (+ 317us) server_negotiation.cc:407] Connection header received
0923 21:05:35.373068 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.373076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.373245 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.373504 (+ 259us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:05:35.373941 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.374878 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.376326 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.376733 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.376842 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.377351 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:05:35.377918 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:05:35.378053 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.378207 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.378247 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260923 21:05:35.384547 19407 tablet.cc:2369] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:35.387881 19621 raft_consensus.cc:1261] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 1 FOLLOWER]: Refusing update from remote peer af946a6fc6da4f6982dbc3fba6e79ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:35.387946 19492 raft_consensus.cc:1261] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 1 FOLLOWER]: Refusing update from remote peer af946a6fc6da4f6982dbc3fba6e79ad5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:35.388939 19698 consensus_queue.cc:1035] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df0802600e0440ab75282455de957e0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:35.389331 19691 consensus_queue.cc:1035] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd97855a99f48948221819977820a9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:35.422066 19220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:35.422470 19220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:35.426653 19220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef9ccaf8e104cecafc30b465011a92e: Sending DeleteTablet for 3 replicas of tablet b3ea324b0472493bb082908a014bd581
I20260923 21:05:35.428082 19602 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ea324b0472493bb082908a014bd581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:35 UTC) from {username='slave'} at 127.0.0.1:59240
I20260923 21:05:35.428082 19472 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ea324b0472493bb082908a014bd581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:35 UTC) from {username='slave'} at 127.0.0.1:40608
I20260923 21:05:35.429149 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19279
I20260923 21:05:35.429890 19341 tablet_service.cc:1514] Processing DeleteTablet for tablet b3ea324b0472493bb082908a014bd581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:35 UTC) from {username='slave'} at 127.0.0.1:54402
I20260923 21:05:35.437230 19714 tablet_replica.cc:330] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: stopping tablet replica
I20260923 21:05:35.437940 19714 raft_consensus.cc:2229] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:05:35.439059 19711 tablet_replica.cc:330] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: stopping tablet replica
I20260923 21:05:35.439504 19712 tablet_replica.cc:330] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: stopping tablet replica
I20260923 21:05:35.440007 19712 raft_consensus.cc:2229] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:35.440083 19711 raft_consensus.cc:2229] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:35.439543 19714 raft_consensus.cc:2258] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:35.440621 19711 raft_consensus.cc:2258] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:35.440693 19712 raft_consensus.cc:2258] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:35.443101 19714 ts_tablet_manager.cc:1905] T b3ea324b0472493bb082908a014bd581 P af946a6fc6da4f6982dbc3fba6e79ad5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:35.443094 19712 ts_tablet_manager.cc:1905] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:35.443341 19711 ts_tablet_manager.cc:1905] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:35.451234 19712 ts_tablet_manager.cc:1918] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:35.451445 19712 log.cc:1196] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/b3ea324b0472493bb082908a014bd581
I20260923 21:05:35.451617 19711 ts_tablet_manager.cc:1918] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:35.451817 19711 log.cc:1196] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/b3ea324b0472493bb082908a014bd581
I20260923 21:05:35.452029 19712 ts_tablet_manager.cc:1939] T b3ea324b0472493bb082908a014bd581 P 8cd97855a99f48948221819977820a9a: Deleting consensus metadata
I20260923 21:05:35.452483 19711 ts_tablet_manager.cc:1939] T b3ea324b0472493bb082908a014bd581 P 0df0802600e0440ab75282455de957e0: Deleting consensus metadata
I20260923 21:05:35.453914 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.430489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.0.97.1:35597)
0923 21:05:35.430679 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.430685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.436971 (+ 6286us) server_negotiation.cc:407] Connection header received
0923 21:05:35.438415 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.438428 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.438645 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.439312 (+ 667us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:35.439848 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.441179 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.448967 (+ 7788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.449958 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.450073 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.451961 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:35.451974 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:35.451997 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:35.452905 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:35.453002 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:35.453033 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:35.453128 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.453421 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.453548 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 21:05:35.454226 19204 catalog_manager.cc:4841] TS 8cd97855a99f48948221819977820a9a (127.0.97.2:36691): tablet b3ea324b0472493bb082908a014bd581 (table test-table [id=da8971ce683445299ae95584e3cea55d]) successfully deleted
I20260923 21:05:35.454679 19206 catalog_manager.cc:4841] TS 0df0802600e0440ab75282455de957e0 (127.0.97.3:33503): tablet b3ea324b0472493bb082908a014bd581 (table test-table [id=da8971ce683445299ae95584e3cea55d]) successfully deleted
I20260923 21:05:35.455206 19401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:35.568471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19279
W20260923 21:05:35.584563 19206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35597 (error 108)
I20260923 21:05:35.585335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19410
I20260923 21:05:35.586609 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.586089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54432)
0923 21:05:35.586309 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.586469 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260923 21:05:35.586889 19206 catalog_manager.cc:4584] TS af946a6fc6da4f6982dbc3fba6e79ad5 (127.0.97.1:35597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3ea324b0472493bb082908a014bd581: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
I20260923 21:05:35.592847 19682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.586177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.97.2:36691)
0923 21:05:35.586843 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.586849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.586869 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:05:35.586936 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.586940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.587094 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.587222 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:35.587650 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.588373 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.589676 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.590093 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.590235 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.591366 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:35.591375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:35.591391 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:35.592023 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:35.592075 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:35.592096 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:35.592230 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.592365 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.592476 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523}
I20260923 21:05:35.593729 19532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:35.641067 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.640415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54446)
0923 21:05:35.640698 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.640824 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 21:05:35.642382 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.641895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54450)
0923 21:05:35.642080 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.642189 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 21:05:35.692533 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19410
I20260923 21:05:35.703261 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.702618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54456)
0923 21:05:35.702881 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.703013 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 21:05:35.704533 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.703995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54458)
0923 21:05:35.704229 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.704338 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 21:05:35.710865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19540
I20260923 21:05:35.718156 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.711755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59242 (local address 127.0.97.3:33503)
0923 21:05:35.711928 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:35.711932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:35.711954 (+ 22us) server_negotiation.cc:407] Connection header received
0923 21:05:35.712216 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:35.712221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:35.712341 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:35.712433 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:35.712884 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.713586 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.715169 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:35.715586 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:35.715689 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:35.716929 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:35.716938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:35.716953 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:35.717547 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:35.717593 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:35.717618 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:35.717703 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:35.717855 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 21:05:35.717938 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 21:05:35.718930 19662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:35.785058 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.784525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54460)
0923 21:05:35.784750 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.784883 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 21:05:35.786231 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:35.785822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35597 (local address 127.0.0.1:54466)
0923 21:05:35.786036 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:35.786109 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 21:05:35.811033 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19540
I20260923 21:05:35.827852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19189
I20260923 21:05:35.829008 19250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:35.941032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19189
2026-09-23T21:05:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2957 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:36 dist-test-slave-8xn5 krb5kdc[19722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:36 dist-test-slave-8xn5 krb5kdc[19722](info): set up 2 sockets
Sep 23 21:05:36 dist-test-slave-8xn5 krb5kdc[19722](info): commencing operation
krb5kdc: starting...
W20260923 21:05:37.099136 388 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:37 dist-test-slave-8xn5 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:05:37.375128 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40963
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:37.716310 19735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:37.716704 19735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:37.716884 19735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:37.724536 19735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:37.724718 19735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:37.724846 19735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:37.724947 19735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:37.725036 19735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:37.735407 19735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:37.736830 19735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:37.738240 19735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:37.750911 19741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:37.751016 19740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:37.750998 19743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:37.752362 19735 server_base.cc:1034] running on GCE node
I20260923 21:05:37.753273 19735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:37.754940 19735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:37.756188 19735 hybrid_clock.cc:648] HybridClock initialized: now 1790197537756138 us; error 46 us; skew 500 ppm
I20260923 21:05:37.756580 19735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:37 dist-test-slave-8xn5 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:37.760001 19735 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:05:37.761690 19735 webserver.cc:460] Webserver started at http://127.0.97.62:41159/ using document root <none> and password file <none>
I20260923 21:05:37.762529 19735 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:37.762670 19735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:37.763005 19735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:37.766217 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "83cad4ae2b194876815bc418d292571e"
format_stamp: "Formatted at 2026-09-23 21:05:37 on dist-test-slave-8xn5"
server_key: "b54c7ccdec42aca4136e5fe6aa371740"
server_key_iv: "8f0595ba55e8883c7c117205e1297a1c"
server_key_version: "encryptionkey@0"
I20260923 21:05:37.767138 19735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "83cad4ae2b194876815bc418d292571e"
format_stamp: "Formatted at 2026-09-23 21:05:37 on dist-test-slave-8xn5"
server_key: "b54c7ccdec42aca4136e5fe6aa371740"
server_key_iv: "8f0595ba55e8883c7c117205e1297a1c"
server_key_version: "encryptionkey@0"
I20260923 21:05:37.773298 19735 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 21:05:37.777583 19750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:37.779421 19735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:05:37.779639 19735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "83cad4ae2b194876815bc418d292571e"
format_stamp: "Formatted at 2026-09-23 21:05:37 on dist-test-slave-8xn5"
server_key: "b54c7ccdec42aca4136e5fe6aa371740"
server_key_iv: "8f0595ba55e8883c7c117205e1297a1c"
server_key_version: "encryptionkey@0"
I20260923 21:05:37.779927 19735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:37.799968 19735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:37.804872 19735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:37.805204 19735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:37.824810 19735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40963
I20260923 21:05:37.824844 19801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40963 every 8 connection(s)
I20260923 21:05:37.826323 19735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:37.830283 19802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:37.831079 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19735
I20260923 21:05:37.831296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:37.831835 388 external_mini_cluster.cc:1428] Setting key 9f6656e7c668868e394475cc801d3d6a
I20260923 21:05:37.845175 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e: Bootstrap starting.
Sep 23 21:05:37 dist-test-slave-8xn5 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:37.849279 19802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:37.850528 19802 log.cc:824] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:37.853658 19802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e: No bootstrap required, opened a new log
I20260923 21:05:37.858539 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:37.833946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.0.97.62:40963)
0923 21:05:37.834914 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:37.834927 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:37.834977 (+ 50us) server_negotiation.cc:407] Connection header received
0923 21:05:37.837102 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:37.837127 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:37.837577 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:37.837964 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:37.839813 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:37.841158 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:37.843090 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:37.843848 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:37.843996 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:37.849633 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:37.849651 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:37.849668 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:37.849729 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:37.853273 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:37.854228 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:37.854236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:37.854243 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:37.854518 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:37.855003 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:37.855009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:37.855012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:37.855709 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:37.856063 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:37.856602 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 21:05:37.857252 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":455,"threads_started":1}
I20260923 21:05:37.864105 19802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad4ae2b194876815bc418d292571e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40963 } }
I20260923 21:05:37.864465 19802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:37.864585 19802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83cad4ae2b194876815bc418d292571e, State: Initialized, Role: FOLLOWER
I20260923 21:05:37.865208 19802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad4ae2b194876815bc418d292571e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40963 } }
I20260923 21:05:37.865449 19802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:37.865583 19802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:37.865769 19802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:37.868264 19802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad4ae2b194876815bc418d292571e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40963 } }
I20260923 21:05:37.868798 19802 leader_election.cc:302] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83cad4ae2b194876815bc418d292571e; no voters:
I20260923 21:05:37.869441 19802 leader_election.cc:288] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:37.869638 19807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:37.870466 19807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [term 1 LEADER]: Becoming Leader. State: Replica: 83cad4ae2b194876815bc418d292571e, State: Running, Role: LEADER
I20260923 21:05:37.871083 19807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad4ae2b194876815bc418d292571e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40963 } }
I20260923 21:05:37.871797 19802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:37.874178 19808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83cad4ae2b194876815bc418d292571e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad4ae2b194876815bc418d292571e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40963 } } }
I20260923 21:05:37.874660 19808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:37.874104 19809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83cad4ae2b194876815bc418d292571e. Latest consensus state: current_term: 1 leader_uuid: "83cad4ae2b194876815bc418d292571e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cad4ae2b194876815bc418d292571e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40963 } } }
I20260923 21:05:37.876041 19809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:37.879988 19814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:37.885720 19814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:37.894976 19814 catalog_manager.cc:1347] Generated new cluster ID: 774781d805ab4fc0a509efd8e55b812c
I20260923 21:05:37.895123 19814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:37.908782 19814 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:37.910207 19814 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:37.919787 19814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e: Generated new TSK 0
I20260923 21:05:37.920733 19814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260923 21:05:37.991827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40963
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:38.346259 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:38.346695 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:38.346869 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:38.354506 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:38.354658 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:38.354851 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:05:38.365566 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:38.367105 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:38.368510 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:38.384411 19834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:38.384424 19836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:38.384270 19833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:38.385185 19828 server_base.cc:1034] running on GCE node
I20260923 21:05:38.385875 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:38.386947 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:38.388216 19828 hybrid_clock.cc:648] HybridClock initialized: now 1790197538388137 us; error 73 us; skew 500 ppm
I20260923 21:05:38.388605 19828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:38 dist-test-slave-8xn5 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:38.392068 19828 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260923 21:05:38.393652 19828 webserver.cc:460] Webserver started at http://127.0.97.1:45973/ using document root <none> and password file <none>
I20260923 21:05:38.394491 19828 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:38.394646 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:38.394989 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:38.398109 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "cc2716c0852f46919e14374af462326d"
format_stamp: "Formatted at 2026-09-23 21:05:38 on dist-test-slave-8xn5"
server_key: "dd0b6321d96abbe7519c03bced272754"
server_key_iv: "0a19ed2ba076146aadbf5841b4d629dd"
server_key_version: "encryptionkey@0"
I20260923 21:05:38.399050 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cc2716c0852f46919e14374af462326d"
format_stamp: "Formatted at 2026-09-23 21:05:38 on dist-test-slave-8xn5"
server_key: "dd0b6321d96abbe7519c03bced272754"
server_key_iv: "0a19ed2ba076146aadbf5841b4d629dd"
server_key_version: "encryptionkey@0"
I20260923 21:05:38.404874 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:05:38.408772 19843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:38.410264 19828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:05:38.410511 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "cc2716c0852f46919e14374af462326d"
format_stamp: "Formatted at 2026-09-23 21:05:38 on dist-test-slave-8xn5"
server_key: "dd0b6321d96abbe7519c03bced272754"
server_key_iv: "0a19ed2ba076146aadbf5841b4d629dd"
server_key_version: "encryptionkey@0"
I20260923 21:05:38.410740 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:38.448853 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:38.453913 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:38.454250 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:38.455396 19828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:38.456833 19828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:38.456951 19828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:38.457106 19828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:38.457181 19828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:38.497040 19828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40103
I20260923 21:05:38.497095 19955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40103 every 8 connection(s)
I20260923 21:05:38.498863 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:05:38.503734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19828
I20260923 21:05:38.503921 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:05:38.504609 388 external_mini_cluster.cc:1428] Setting key f721490bf34091cd7bb62996c70d0d7e
Sep 23 21:05:38 dist-test-slave-8xn5 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:38.525813 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:38.502116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33999 (local address 127.0.97.62:40963)
0923 21:05:38.502676 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:38.502684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:38.504078 (+ 1394us) server_negotiation.cc:407] Connection header received
0923 21:05:38.506612 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:38.506621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:38.506740 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:38.506851 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:38.508515 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:38.509852 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:38.511803 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:38.512624 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:38.512828 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:38.517414 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:38.517425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:38.517433 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:38.517486 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:38.521570 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:38.522779 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:38.522789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:38.522795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:38.522966 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:38.523557 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:38.523565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:38.523571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:38.523988 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:38.524150 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:38.525038 (+ 888us) server_negotiation.cc:299] Negotiation successful
0923 21:05:38.525331 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
I20260923 21:05:38.526708 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:38.502760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40963 (local address 127.0.97.1:33999)
0923 21:05:38.503891 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:38.503959 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:38.506178 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:38.507138 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:38.507148 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:38.507715 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:38.508307 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:38.508326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:38.510175 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:38.510180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:38.511621 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:38.511636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:38.512915 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:38.512921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:38.513287 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:38.514231 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:38.514256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:38.517100 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:38.521794 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:38.521804 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:38.521820 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:38.522611 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:38.523171 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:38.523175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:38.523178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:38.523418 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:38.524213 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:38.524220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:38.524590 (+ 370us) client_negotiation.cc:769] Sending connection context
0923 21:05:38.524848 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 21:05:38.525322 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":570,"threads_started":1}
I20260923 21:05:38.528544 19956 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40963
I20260923 21:05:38.528913 19956 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:38.529829 19956 heartbeater.cc:511] Master 127.0.97.62:40963 requested a full tablet report, sending...
I20260923 21:05:38.532375 19767 ts_manager.cc:194] Registered new tserver with Master: cc2716c0852f46919e14374af462326d (127.0.97.1:40103)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260923 21:05:38.534569 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260923 21:05:38.566613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40963
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:38.907742 19962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:38.908046 19962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:38.908195 19962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:38.915966 19962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:38.916133 19962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:38.916322 19962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:05:38.926534 19962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:38.927906 19962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:38.929277 19962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:38.943307 19967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:38.943341 19970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:38.943956 19968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:38.944700 19962 server_base.cc:1034] running on GCE node
I20260923 21:05:38.945385 19962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:38.946621 19962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:38.947865 19962 hybrid_clock.cc:648] HybridClock initialized: now 1790197538947812 us; error 47 us; skew 500 ppm
I20260923 21:05:38.948216 19962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:38 dist-test-slave-8xn5 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:38.951664 19962 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260923 21:05:38.953405 19962 webserver.cc:460] Webserver started at http://127.0.97.2:44263/ using document root <none> and password file <none>
I20260923 21:05:38.954144 19962 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:38.954319 19962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:38.954715 19962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:38.957752 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "1343cb61e19d441585b517157ec08b20"
format_stamp: "Formatted at 2026-09-23 21:05:38 on dist-test-slave-8xn5"
server_key: "b4f396f50bb3ca683b7b123dbba460c4"
server_key_iv: "e938235805fb5476681e8524b0d1e8c6"
server_key_version: "encryptionkey@0"
I20260923 21:05:38.958673 19962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1343cb61e19d441585b517157ec08b20"
format_stamp: "Formatted at 2026-09-23 21:05:38 on dist-test-slave-8xn5"
server_key: "b4f396f50bb3ca683b7b123dbba460c4"
server_key_iv: "e938235805fb5476681e8524b0d1e8c6"
server_key_version: "encryptionkey@0"
I20260923 21:05:38.964464 19962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:05:38.968436 19977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:38.970031 19962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:05:38.970255 19962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "1343cb61e19d441585b517157ec08b20"
format_stamp: "Formatted at 2026-09-23 21:05:38 on dist-test-slave-8xn5"
server_key: "b4f396f50bb3ca683b7b123dbba460c4"
server_key_iv: "e938235805fb5476681e8524b0d1e8c6"
server_key_version: "encryptionkey@0"
I20260923 21:05:38.970541 19962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:38.989076 19962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:38.993779 19962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:38.994122 19962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:38.995357 19962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:38.996760 19962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:38.996868 19962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:38.997026 19962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:38.997102 19962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:39.034493 19962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43623
I20260923 21:05:39.034528 20089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43623 every 8 connection(s)
I20260923 21:05:39.035807 19962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:05:39.044505 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 19962
I20260923 21:05:39.044708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:05:39.045154 388 external_mini_cluster.cc:1428] Setting key 9ed9bcdf2199e04211513817918e4aee
Sep 23 21:05:39 dist-test-slave-8xn5 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:39.056783 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.038404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51267 (local address 127.0.97.62:40963)
0923 21:05:39.038855 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.038859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.039765 (+ 906us) server_negotiation.cc:407] Connection header received
0923 21:05:39.041642 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.041648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.041744 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.041868 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:39.043491 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.044209 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.045996 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.046651 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.046799 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.050667 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:39.050672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:39.050676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:39.050713 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:39.053466 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:39.054289 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:39.054294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:39.054296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:39.054485 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:39.054955 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:39.054960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:39.054964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:39.055253 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:39.055415 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.056165 (+ 750us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.056401 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260923 21:05:39.057845 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.038868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40963 (local address 127.0.97.2:51267)
0923 21:05:39.039585 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:39.039625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:39.041358 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:39.042118 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:39.042126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:39.042734 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:39.043315 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.043332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.044371 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.044384 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.045834 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.045848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.046909 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.046918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.047167 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.047910 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:39.047935 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:39.050324 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:39.053641 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:39.053660 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:39.053684 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:39.054140 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:39.054606 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:39.054612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:39.054620 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:39.054831 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:39.055389 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:39.055397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:39.055776 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 21:05:39.056055 (+ 279us) client_negotiation.cc:240] Negotiation successful
0923 21:05:39.056630 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":286,"threads_started":1}
I20260923 21:05:39.059504 20090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40963
I20260923 21:05:39.059854 20090 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:39.060829 20090 heartbeater.cc:511] Master 127.0.97.62:40963 requested a full tablet report, sending...
I20260923 21:05:39.062589 19767 ts_manager.cc:194] Registered new tserver with Master: 1343cb61e19d441585b517157ec08b20 (127.0.97.2:43623)
I20260923 21:05:39.063911 19767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51267
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260923 21:05:39.106305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40963
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:39.462615 20096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:39.462915 20096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:39.463017 20096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:39.470670 20096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:39.470798 20096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:39.470912 20096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:05:39.481359 20096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:39.482947 20096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:39.484390 20096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:39.498811 20101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:39.498847 20102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:39.499750 20104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:39.499881 20096 server_base.cc:1034] running on GCE node
I20260923 21:05:39.500708 20096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:39.502146 20096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:39.503418 20096 hybrid_clock.cc:648] HybridClock initialized: now 1790197539503328 us; error 81 us; skew 500 ppm
I20260923 21:05:39.503839 20096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:39 dist-test-slave-8xn5 krb5kdc[19722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:39.507640 20096 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260923 21:05:39.509343 20096 webserver.cc:460] Webserver started at http://127.0.97.3:44835/ using document root <none> and password file <none>
I20260923 21:05:39.510088 20096 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:39.510192 20096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:39.510545 20096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:39.513634 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "42aee42f451c4bffaf2cc72bf01005f0"
format_stamp: "Formatted at 2026-09-23 21:05:39 on dist-test-slave-8xn5"
server_key: "5f3e2454c7d360da182c708d9c018339"
server_key_iv: "6c15cefeebfe3577a6e27e69f715e8ad"
server_key_version: "encryptionkey@0"
I20260923 21:05:39.514626 20096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "42aee42f451c4bffaf2cc72bf01005f0"
format_stamp: "Formatted at 2026-09-23 21:05:39 on dist-test-slave-8xn5"
server_key: "5f3e2454c7d360da182c708d9c018339"
server_key_iv: "6c15cefeebfe3577a6e27e69f715e8ad"
server_key_version: "encryptionkey@0"
I20260923 21:05:39.520844 20096 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260923 21:05:39.525645 20111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:39.527578 20096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:05:39.527798 20096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "42aee42f451c4bffaf2cc72bf01005f0"
format_stamp: "Formatted at 2026-09-23 21:05:39 on dist-test-slave-8xn5"
server_key: "5f3e2454c7d360da182c708d9c018339"
server_key_iv: "6c15cefeebfe3577a6e27e69f715e8ad"
server_key_version: "encryptionkey@0"
I20260923 21:05:39.528035 20096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:39.538756 19956 heartbeater.cc:503] Master 127.0.97.62:40963 was elected leader, sending a full tablet report...
I20260923 21:05:39.564776 20096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:39.570055 20096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:39.570446 20096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:39.571579 20096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:39.573143 20096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:39.573253 20096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:39.573441 20096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:39.573551 20096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:39.611217 20096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40547
I20260923 21:05:39.611266 20223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40547 every 8 connection(s)
I20260923 21:05:39.612746 20096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:05:39.614485 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 20096
I20260923 21:05:39.614666 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:05:39.615226 388 external_mini_cluster.cc:1428] Setting key 75140e7eedf94af032065aa7b62ba913
Sep 23 21:05:39 dist-test-slave-8xn5 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:05:39.636376 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.615990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60679 (local address 127.0.97.62:40963)
0923 21:05:39.616508 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.616516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.617807 (+ 1291us) server_negotiation.cc:407] Connection header received
0923 21:05:39.620079 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.620083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.620215 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.620324 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:39.622131 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.623069 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.624924 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.625342 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.625447 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.629797 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:39.629803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:39.629806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:39.629844 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:39.632905 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:39.633857 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:39.633865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:39.633870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:39.634010 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:39.634536 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:39.634541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:39.634544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:39.634827 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:39.634988 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.635744 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.635978 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":435,"threads_started":1}
I20260923 21:05:39.637394 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.616713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40963 (local address 127.0.97.3:60679)
0923 21:05:39.617620 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:39.617672 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:39.619770 (+ 2098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:39.620667 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:39.620677 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:39.621258 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:39.621888 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.621907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.623278 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.623282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.624751 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.624766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.625621 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.625627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.625870 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.626798 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:05:39.626825 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:39.629542 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:39.633103 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:39.633112 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:39.633131 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:39.633725 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:39.634128 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:05:39.634133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:05:39.634137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:05:39.634408 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:05:39.634961 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:39.634969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:39.635365 (+ 396us) client_negotiation.cc:769] Sending connection context
0923 21:05:39.635645 (+ 280us) client_negotiation.cc:240] Negotiation successful
0923 21:05:39.636234 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":391,"threads_started":1}
I20260923 21:05:39.639026 20224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40963
I20260923 21:05:39.639387 20224 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:39.640194 20224 heartbeater.cc:511] Master 127.0.97.62:40963 requested a full tablet report, sending...
I20260923 21:05:39.642207 19766 ts_manager.cc:194] Registered new tserver with Master: 42aee42f451c4bffaf2cc72bf01005f0 (127.0.97.3:40547)
I20260923 21:05:39.643282 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:05:39.643404 19766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60679
I20260923 21:05:39.660702 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.647666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.0.97.62:40963)
0923 21:05:39.647841 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.647845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.648226 (+ 381us) server_negotiation.cc:407] Connection header received
0923 21:05:39.648333 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.648337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.648405 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.648537 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:39.649978 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.650929 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.652201 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.652692 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.652827 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.655249 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:39.655254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:39.655258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:39.655293 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:39.658269 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:39.658948 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:39.658952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:39.658955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:39.659083 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:39.659437 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:39.659441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:39.659451 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:39.659823 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:39.660112 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.660272 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.660454 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 21:05:39.664883 19766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:05:39.672147 19766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:05:39.714932 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.686603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.0.97.3:40547)
0923 21:05:39.703527 (+ 16924us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.703551 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.703604 (+ 53us) server_negotiation.cc:407] Connection header received
0923 21:05:39.703748 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.703760 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.704114 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.704428 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:39.705548 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.707897 (+ 2349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.712375 (+ 4478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.713653 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.713810 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.714071 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.714398 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.714571 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16677,"thread_start_us":16463,"threads_started":1}
I20260923 21:05:39.715482 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.686254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40547 (local address 127.0.0.1:32912)
0923 21:05:39.693754 (+ 7500us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:39.693815 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:39.694085 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:39.704528 (+ 10443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:39.704542 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:39.704613 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:39.705232 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.705252 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.708185 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.708195 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.712162 (+ 3967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.712177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.713871 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.713877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.714094 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.714165 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 21:05:39.714309 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 21:05:39.714580 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7116,"thread_start_us":751,"threads_started":1}
I20260923 21:05:39.719354 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.687204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40103 (local address 127.0.0.1:59524)
0923 21:05:39.710499 (+ 23295us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:39.710516 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:39.710827 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:39.711971 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:39.711975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:39.712000 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:39.712500 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.712511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.714536 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.714541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.716932 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.716948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.718438 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.718444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.718623 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.718678 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 21:05:39.718775 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 21:05:39.718917 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23165,"mutex_wait_us":2,"thread_start_us":23666,"threads_started":1}
I20260923 21:05:39.719368 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.687492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.0.97.1:40103)
0923 21:05:39.709703 (+ 22211us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.709715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.710653 (+ 938us) server_negotiation.cc:407] Connection header received
0923 21:05:39.711120 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.711130 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.711441 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.711701 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:39.712696 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.714371 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.717141 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.718207 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.718388 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.718640 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.718839 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.719000 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22025,"thread_start_us":21868,"threads_started":1}
I20260923 21:05:39.720140 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.687827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43623 (local address 127.0.0.1:37244)
0923 21:05:39.711311 (+ 23484us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:39.711329 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:39.711446 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:39.712370 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:39.712374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:39.712406 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:39.712869 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.712880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.714877 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.714887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.717155 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.717165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.719364 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.719372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.719561 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.719596 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:05:39.719687 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 21:05:39.719816 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23343,"thread_start_us":22276,"threads_started":1}
I20260923 21:05:39.721239 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.688384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.0.97.2:43623)
0923 21:05:39.708795 (+ 20411us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.708809 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.711587 (+ 2778us) server_negotiation.cc:407] Connection header received
0923 21:05:39.711703 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.711709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.711985 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.712207 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:39.713053 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.714703 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.717372 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.718862 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.718988 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.719200 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.720729 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.720917 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20119,"thread_start_us":20437,"threads_started":1}
I20260923 21:05:39.721452 20159 tablet_service.cc:1467] Processing CreateTablet for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c (DEFAULT_TABLE table=test-table [id=679193bc7e5a426195bb66b59a28f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:39.723199 20159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:39.723713 19891 tablet_service.cc:1467] Processing CreateTablet for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c (DEFAULT_TABLE table=test-table [id=679193bc7e5a426195bb66b59a28f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:39.724187 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c (DEFAULT_TABLE table=test-table [id=679193bc7e5a426195bb66b59a28f701]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:39.725220 19891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:39.725519 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:39.737044 20243 tablet_bootstrap.cc:492] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Bootstrap starting.
I20260923 21:05:39.740841 20245 tablet_bootstrap.cc:492] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Bootstrap starting.
I20260923 21:05:39.741994 20243 tablet_bootstrap.cc:654] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:39.743001 20244 tablet_bootstrap.cc:492] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Bootstrap starting.
I20260923 21:05:39.743564 20243 log.cc:824] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:39.744618 20245 tablet_bootstrap.cc:654] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:39.745975 20245 log.cc:824] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:39.746639 20244 tablet_bootstrap.cc:654] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:39.747241 20243 tablet_bootstrap.cc:492] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: No bootstrap required, opened a new log
I20260923 21:05:39.747558 20243 ts_tablet_manager.cc:1397] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260923 21:05:39.748013 20244 log.cc:824] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:39.750121 20245 tablet_bootstrap.cc:492] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: No bootstrap required, opened a new log
I20260923 21:05:39.750442 20245 ts_tablet_manager.cc:1397] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 21:05:39.757012 20244 tablet_bootstrap.cc:492] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: No bootstrap required, opened a new log
I20260923 21:05:39.757333 20244 ts_tablet_manager.cc:1397] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260923 21:05:39.758625 20243 raft_consensus.cc:348] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.759285 20243 raft_consensus.cc:374] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:39.759423 20243 raft_consensus.cc:729] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1343cb61e19d441585b517157ec08b20, State: Initialized, Role: FOLLOWER
I20260923 21:05:39.760247 20243 consensus_queue.cc:260] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.761513 20090 heartbeater.cc:503] Master 127.0.97.62:40963 was elected leader, sending a full tablet report...
I20260923 21:05:39.762195 20243 ts_tablet_manager.cc:1428] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260923 21:05:39.761902 20245 raft_consensus.cc:348] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.762406 20245 raft_consensus.cc:374] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:39.762559 20245 raft_consensus.cc:729] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2716c0852f46919e14374af462326d, State: Initialized, Role: FOLLOWER
I20260923 21:05:39.763254 20245 consensus_queue.cc:260] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.765123 20245 ts_tablet_manager.cc:1428] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260923 21:05:39.768697 20244 raft_consensus.cc:348] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.769152 20244 raft_consensus.cc:374] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:39.769284 20244 raft_consensus.cc:729] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42aee42f451c4bffaf2cc72bf01005f0, State: Initialized, Role: FOLLOWER
I20260923 21:05:39.769876 20244 consensus_queue.cc:260] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.771090 20224 heartbeater.cc:503] Master 127.0.97.62:40963 was elected leader, sending a full tablet report...
I20260923 21:05:39.772177 20244 ts_tablet_manager.cc:1428] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
W20260923 21:05:39.787842 20091 tablet.cc:2369] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:39.864682 20225 tablet.cc:2369] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:39.936097 20251 raft_consensus.cc:482] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:39.936434 20251 raft_consensus.cc:504] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.938309 20251 leader_election.cc:288] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1343cb61e19d441585b517157ec08b20 (127.0.97.2:43623), cc2716c0852f46919e14374af462326d (127.0.97.1:40103)
I20260923 21:05:39.945667 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.939104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44527 (local address 127.0.97.1:40103)
0923 21:05:39.939305 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.939311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.939334 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:05:39.939514 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.939523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.939616 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.939721 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:39.940266 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.941273 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.943310 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.944626 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.944746 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.944828 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.945274 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.945376 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:05:39.945719 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.938720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40103 (local address 127.0.97.3:44527)
0923 21:05:39.939131 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:39.939157 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:39.939307 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:39.939905 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:39.939909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:39.939937 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:39.940104 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.940112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.941432 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.941439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.943129 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.943141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.944823 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.944829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.945025 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.945065 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 21:05:39.945239 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 21:05:39.945389 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260923 21:05:39.946599 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.939174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46463 (local address 127.0.97.2:43623)
0923 21:05:39.939424 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:39.939433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:39.940058 (+ 625us) server_negotiation.cc:407] Connection header received
0923 21:05:39.940321 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:39.940325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:39.940391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:39.940465 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:39.941079 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.942142 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.944298 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.945373 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.945509 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.945609 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:39.946175 (+ 566us) server_negotiation.cc:299] Negotiation successful
0923 21:05:39.946317 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 21:05:39.946627 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:39.938869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43623 (local address 127.0.97.3:46463)
0923 21:05:39.939936 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:39.939951 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:39.940118 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:39.940518 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:39.940523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:39.940549 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:39.940899 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.940912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.942309 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.942312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.944128 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:39.944143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:39.945722 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:39.945731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:39.945943 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:39.945973 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 21:05:39.946087 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 21:05:39.946220 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":74,"thread_start_us":683,"threads_started":1}
I20260923 21:05:39.947000 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1ad668b1b4e0aa3533b6b4f3c365c" candidate_uuid: "42aee42f451c4bffaf2cc72bf01005f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc2716c0852f46919e14374af462326d" is_pre_election: true
I20260923 21:05:39.947703 19911 raft_consensus.cc:2454] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42aee42f451c4bffaf2cc72bf01005f0 in term 0.
I20260923 21:05:39.947813 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1ad668b1b4e0aa3533b6b4f3c365c" candidate_uuid: "42aee42f451c4bffaf2cc72bf01005f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1343cb61e19d441585b517157ec08b20" is_pre_election: true
I20260923 21:05:39.948346 20045 raft_consensus.cc:2454] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42aee42f451c4bffaf2cc72bf01005f0 in term 0.
I20260923 21:05:39.948467 20112 leader_election.cc:302] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42aee42f451c4bffaf2cc72bf01005f0, cc2716c0852f46919e14374af462326d; no voters:
I20260923 21:05:39.948938 20251 raft_consensus.cc:2781] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:05:39.949147 20251 raft_consensus.cc:482] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:05:39.949285 20251 raft_consensus.cc:3037] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:39.951694 20251 raft_consensus.cc:504] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.952369 20251 leader_election.cc:288] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [CANDIDATE]: Term 1 election: Requested vote from peers 1343cb61e19d441585b517157ec08b20 (127.0.97.2:43623), cc2716c0852f46919e14374af462326d (127.0.97.1:40103)
I20260923 21:05:39.952991 19911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1ad668b1b4e0aa3533b6b4f3c365c" candidate_uuid: "42aee42f451c4bffaf2cc72bf01005f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc2716c0852f46919e14374af462326d"
I20260923 21:05:39.953039 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af1ad668b1b4e0aa3533b6b4f3c365c" candidate_uuid: "42aee42f451c4bffaf2cc72bf01005f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1343cb61e19d441585b517157ec08b20"
I20260923 21:05:39.953318 20045 raft_consensus.cc:3037] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:39.953331 19911 raft_consensus.cc:3037] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:39.955751 19911 raft_consensus.cc:2454] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42aee42f451c4bffaf2cc72bf01005f0 in term 1.
I20260923 21:05:39.955752 20045 raft_consensus.cc:2454] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42aee42f451c4bffaf2cc72bf01005f0 in term 1.
I20260923 21:05:39.956305 20112 leader_election.cc:302] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42aee42f451c4bffaf2cc72bf01005f0, cc2716c0852f46919e14374af462326d; no voters:
I20260923 21:05:39.956673 20251 raft_consensus.cc:2781] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:39.957173 20251 raft_consensus.cc:686] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 1 LEADER]: Becoming Leader. State: Replica: 42aee42f451c4bffaf2cc72bf01005f0, State: Running, Role: LEADER
I20260923 21:05:39.957665 20251 consensus_queue.cc:237] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } }
I20260923 21:05:39.962878 19766 catalog_manager.cc:5495] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42aee42f451c4bffaf2cc72bf01005f0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "42aee42f451c4bffaf2cc72bf01005f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42aee42f451c4bffaf2cc72bf01005f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 } health_report { overall_health: UNKNOWN } } }
W20260923 21:05:40.001842 19957 tablet.cc:2369] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:40.019193 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:40.013243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.97.3:40547)
0923 21:05:40.013508 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:40.013524 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:40.013634 (+ 110us) server_negotiation.cc:407] Connection header received
0923 21:05:40.013938 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:40.013942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:40.014029 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:40.014241 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:05:40.014667 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.015389 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.016836 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.017258 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.017406 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:40.017958 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:05:40.018534 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:05:40.018706 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:40.018816 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 21:05:40.018907 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260923 21:05:40.028347 20045 raft_consensus.cc:1261] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 1 FOLLOWER]: Refusing update from remote peer 42aee42f451c4bffaf2cc72bf01005f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:40.028347 19911 raft_consensus.cc:1261] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 1 FOLLOWER]: Refusing update from remote peer 42aee42f451c4bffaf2cc72bf01005f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:40.029387 20253 consensus_queue.cc:1035] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1343cb61e19d441585b517157ec08b20" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:40.029739 20251 consensus_queue.cc:1035] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc2716c0852f46919e14374af462326d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:40.050632 20256 mvcc.cc:205] Tried to move back new op lower bound from 7332649123945361408 to 7332649123670839296. Current Snapshot: MvccSnapshot[applied={T|T < 7332649123945361408}]
I20260923 21:05:40.051438 20257 mvcc.cc:205] Tried to move back new op lower bound from 7332649123945361408 to 7332649123670839296. Current Snapshot: MvccSnapshot[applied={T|T < 7332649123945361408}]
I20260923 21:05:40.057663 20258 mvcc.cc:205] Tried to move back new op lower bound from 7332649123945361408 to 7332649123670839296. Current Snapshot: MvccSnapshot[applied={T|T < 7332649123945361408}]
I20260923 21:05:40.078529 19767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:40.078852 19767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:40.082993 19767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83cad4ae2b194876815bc418d292571e: Sending DeleteTablet for 3 replicas of tablet 5af1ad668b1b4e0aa3533b6b4f3c365c
I20260923 21:05:40.084587 20025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37244
I20260923 21:05:40.084616 19891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59524
I20260923 21:05:40.084641 20159 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af1ad668b1b4e0aa3533b6b4f3c365c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32912
I20260923 21:05:40.086685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19828
I20260923 21:05:40.096427 20267 tablet_replica.cc:330] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: stopping tablet replica
I20260923 21:05:40.097329 20267 raft_consensus.cc:2229] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:40.098179 20267 raft_consensus.cc:2258] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:40.098860 20268 tablet_replica.cc:330] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: stopping tablet replica
I20260923 21:05:40.098899 20269 tablet_replica.cc:330] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: stopping tablet replica
I20260923 21:05:40.099373 20268 raft_consensus.cc:2229] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:40.099927 20268 raft_consensus.cc:2258] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:40.099429 20269 raft_consensus.cc:2229] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:05:40.101128 20267 ts_tablet_manager.cc:1905] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:40.101414 20269 raft_consensus.cc:2258] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:40.103891 20269 ts_tablet_manager.cc:1905] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 21:05:40 dist-test-slave-8xn5 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260923 21:05:40.111287 20267 ts_tablet_manager.cc:1918] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:40.103358 20268 ts_tablet_manager.cc:1905] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:40.112272 20269 ts_tablet_manager.cc:1918] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:40.112563 20269 log.cc:1196] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/5af1ad668b1b4e0aa3533b6b4f3c365c
I20260923 21:05:40.113371 20269 ts_tablet_manager.cc:1939] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 42aee42f451c4bffaf2cc72bf01005f0: Deleting consensus metadata
I20260923 21:05:40.115672 19754 catalog_manager.cc:4841] TS 42aee42f451c4bffaf2cc72bf01005f0 (127.0.97.3:40547): tablet 5af1ad668b1b4e0aa3533b6b4f3c365c (table test-table [id=679193bc7e5a426195bb66b59a28f701]) successfully deleted
I20260923 21:05:40.117098 20267 log.cc:1196] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/5af1ad668b1b4e0aa3533b6b4f3c365c
I20260923 21:05:40.117764 20267 ts_tablet_manager.cc:1939] T 5af1ad668b1b4e0aa3533b6b4f3c365c P 1343cb61e19d441585b517157ec08b20: Deleting consensus metadata
I20260923 21:05:40.119586 20268 ts_tablet_manager.cc:1918] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:40.119830 20268 log.cc:1196] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790197207951953-388-0/minicluster-data/ts-0/wal/wals/5af1ad668b1b4e0aa3533b6b4f3c365c
I20260923 21:05:40.120028 19754 catalog_manager.cc:4841] TS 1343cb61e19d441585b517157ec08b20 (127.0.97.2:43623): tablet 5af1ad668b1b4e0aa3533b6b4f3c365c (table test-table [id=679193bc7e5a426195bb66b59a28f701]) successfully deleted
I20260923 21:05:40.120294 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:40.088190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.0.97.1:40103)
0923 21:05:40.088442 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:40.088454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:40.093417 (+ 4963us) server_negotiation.cc:407] Connection header received
0923 21:05:40.093681 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:40.093691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:40.093836 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:40.093979 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:40.100790 (+ 6811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.102018 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.104069 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.104737 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.104847 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:40.110750 (+ 5903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:40.110772 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:40.110792 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:40.110848 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:40.115171 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:40.116280 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:40.116292 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:40.116296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:40.116492 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:40.117010 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:40.117017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:40.117027 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:40.117655 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:40.118022 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:40.119344 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0923 21:05:40.119651 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 21:05:40.120756 20268 ts_tablet_manager.cc:1939] T 5af1ad668b1b4e0aa3533b6b4f3c365c P cc2716c0852f46919e14374af462326d: Deleting consensus metadata
I20260923 21:05:40.121248 19951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:40.257520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19828
I20260923 21:05:40.257694 19751 catalog_manager.cc:4841] TS cc2716c0852f46919e14374af462326d (127.0.97.1:40103): tablet 5af1ad668b1b4e0aa3533b6b4f3c365c (table test-table [id=679193bc7e5a426195bb66b59a28f701]) successfully deleted
I20260923 21:05:40.276258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19962
Sep 23 21:05:40 dist-test-slave-8xn5 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260923 21:05:40.295440 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:40.277401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.97.2:43623)
0923 21:05:40.277702 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:40.277711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:40.277734 (+ 23us) server_negotiation.cc:407] Connection header received
0923 21:05:40.277791 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:40.277794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:40.277871 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:40.277992 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:40.279592 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.280402 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.281884 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.282459 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.282623 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:40.288065 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:40.288083 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:40.288100 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:40.288143 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:40.292071 (+ 3928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:40.292983 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:40.292992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:40.292995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:40.293149 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:40.293573 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:40.293578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:40.293581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:40.294170 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:40.294529 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:40.294916 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 21:05:40.295119 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260923 21:05:40.296079 20085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:40.391240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19962
I20260923 21:05:40.416280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20096
Sep 23 21:05:40 dist-test-slave-8xn5 krb5kdc[19722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260923 21:05:40.436833 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:40.417202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.0.97.3:40547)
0923 21:05:40.417382 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:40.417386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:40.417458 (+ 72us) server_negotiation.cc:407] Connection header received
0923 21:05:40.417700 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:40.417704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:40.417784 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:40.417872 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:40.419982 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.420834 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.422897 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:40.423569 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:40.423711 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:40.429971 (+ 6260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:40.429989 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:40.430009 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:40.430069 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:40.433702 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:40.434614 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:40.434624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:40.434626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:40.434780 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:40.435198 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:40.435202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:40.435203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:40.435585 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:40.435905 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:40.436224 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 21:05:40.436469 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:05:40.437564 20219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:40.563200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 20096
I20260923 21:05:40.583931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 19735
I20260923 21:05:40.585142 19797 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:40.698575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 19735
2026-09-23T21:05:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4756 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T21:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:40Z Disabled control of system clock
I20260923 21:05:40.778795 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39041
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 21:05:41.127354 20278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:41.127683 20278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:41.127767 20278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:41.135437 20278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:41.135564 20278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:41.135605 20278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:41.135658 20278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:41.135703 20278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:41.145129 20278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:41.146458 20278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:41.148317 20278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:41.161326 20283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:41.161414 20286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:41.161381 20284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:41.162446 20278 server_base.cc:1034] running on GCE node
I20260923 21:05:41.163347 20278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:41.164801 20278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:41.166005 20278 hybrid_clock.cc:648] HybridClock initialized: now 1790197541165938 us; error 63 us; skew 500 ppm
I20260923 21:05:41.166421 20278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:41.174705 20278 webserver.cc:460] Webserver started at http://127.0.97.62:36649/ using document root <none> and password file <none>
I20260923 21:05:41.175487 20278 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:41.175664 20278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:41.175971 20278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:41.179208 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "def08aac7d2f400ab4468c65caecb980"
format_stamp: "Formatted at 2026-09-23 21:05:41 on dist-test-slave-8xn5"
server_key: "1b441964f1f847687d61838606fcc89c"
server_key_iv: "ca4c5006b0d47a12d71f462d700fb660"
server_key_version: "encryptionkey@0"
I20260923 21:05:41.180028 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "def08aac7d2f400ab4468c65caecb980"
format_stamp: "Formatted at 2026-09-23 21:05:41 on dist-test-slave-8xn5"
server_key: "1b441964f1f847687d61838606fcc89c"
server_key_iv: "ca4c5006b0d47a12d71f462d700fb660"
server_key_version: "encryptionkey@0"
I20260923 21:05:41.185988 20278 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260923 21:05:41.190402 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:41.192739 20278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:05:41.193006 20278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "def08aac7d2f400ab4468c65caecb980"
format_stamp: "Formatted at 2026-09-23 21:05:41 on dist-test-slave-8xn5"
server_key: "1b441964f1f847687d61838606fcc89c"
server_key_iv: "ca4c5006b0d47a12d71f462d700fb660"
server_key_version: "encryptionkey@0"
I20260923 21:05:41.193343 20278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:41.212405 20278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:41.213362 20278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:41.213639 20278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:41.236153 20278 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39041
I20260923 21:05:41.236182 20343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39041 every 8 connection(s)
I20260923 21:05:41.237831 20278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:41.242506 20344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:41.247314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 20278
I20260923 21:05:41.247534 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:41.248059 388 external_mini_cluster.cc:1428] Setting key 316e334edbd26d42574ba9ac2cd6e2b6
I20260923 21:05:41.257169 20344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980: Bootstrap starting.
I20260923 21:05:41.261454 20344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:41.263002 20344 log.cc:824] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:41.267141 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:41.250284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.0.97.62:39041)
0923 21:05:41.251700 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:41.251720 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:41.251791 (+ 71us) server_negotiation.cc:407] Connection header received
0923 21:05:41.254900 (+ 3109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:41.254946 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:41.255717 (+ 771us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:41.256252 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:41.256888 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:41.259217 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:41.260759 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:41.261994 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:41.262194 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:41.263700 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:41.263717 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:41.263744 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:41.264418 (+ 674us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:41.264578 (+ 160us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:41.264610 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:41.264748 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:41.265189 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 21:05:41.265636 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":408,"threads_started":1}
I20260923 21:05:41.267246 20344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980: No bootstrap required, opened a new log
I20260923 21:05:41.276261 20344 raft_consensus.cc:348] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def08aac7d2f400ab4468c65caecb980" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39041 } }
I20260923 21:05:41.276722 20344 raft_consensus.cc:374] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:41.276821 20344 raft_consensus.cc:729] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def08aac7d2f400ab4468c65caecb980, State: Initialized, Role: FOLLOWER
I20260923 21:05:41.277493 20344 consensus_queue.cc:260] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def08aac7d2f400ab4468c65caecb980" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39041 } }
I20260923 21:05:41.277735 20344 raft_consensus.cc:388] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:41.277845 20344 raft_consensus.cc:482] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:41.278003 20344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:41.280527 20344 raft_consensus.cc:504] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def08aac7d2f400ab4468c65caecb980" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39041 } }
I20260923 21:05:41.281090 20344 leader_election.cc:302] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: def08aac7d2f400ab4468c65caecb980; no voters:
I20260923 21:05:41.281766 20344 leader_election.cc:288] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:41.282253 20349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:41.283418 20349 raft_consensus.cc:686] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [term 1 LEADER]: Becoming Leader. State: Replica: def08aac7d2f400ab4468c65caecb980, State: Running, Role: LEADER
I20260923 21:05:41.284063 20349 consensus_queue.cc:237] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def08aac7d2f400ab4468c65caecb980" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39041 } }
I20260923 21:05:41.284580 20344 sys_catalog.cc:564] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:41.287245 20350 sys_catalog.cc:455] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "def08aac7d2f400ab4468c65caecb980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def08aac7d2f400ab4468c65caecb980" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39041 } } }
I20260923 21:05:41.287700 20350 sys_catalog.cc:458] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:41.287806 20351 sys_catalog.cc:455] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader def08aac7d2f400ab4468c65caecb980. Latest consensus state: current_term: 1 leader_uuid: "def08aac7d2f400ab4468c65caecb980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def08aac7d2f400ab4468c65caecb980" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39041 } } }
I20260923 21:05:41.288343 20351 sys_catalog.cc:458] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:41.291836 20358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:41.298146 20358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:41.309666 20358 catalog_manager.cc:1347] Generated new cluster ID: 64b484db12864218a6b7c224ebc99e86
I20260923 21:05:41.309846 20358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:41.364367 20358 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:41.365995 20358 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:41.374065 20358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980: Generated new TSK 0
I20260923 21:05:41.375160 20358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:41.388449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39041
--builtin_ntp_servers=127.0.97.20:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 21:05:41.746934 20368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:41.747247 20368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:41.747421 20368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:41.755288 20368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:41.755402 20368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:41.755581 20368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260923 21:05:41.765291 20368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:41.767066 20368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:41.769254 20368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:41.783668 20373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:41.783699 20374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:41.783886 20376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:41.784760 20368 server_base.cc:1034] running on GCE node
I20260923 21:05:41.785447 20368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:41.786577 20368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:41.787858 20368 hybrid_clock.cc:648] HybridClock initialized: now 1790197541787800 us; error 49 us; skew 500 ppm
I20260923 21:05:41.788347 20368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:41.791412 20368 webserver.cc:460] Webserver started at http://127.0.97.1:33829/ using document root <none> and password file <none>
I20260923 21:05:41.792271 20368 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:41.792407 20368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:41.792804 20368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:41.796278 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data/instance:
uuid: "3934f642b43c4a18b6e60d6563f1d902"
format_stamp: "Formatted at 2026-09-23 21:05:41 on dist-test-slave-8xn5"
server_key: "bbd1ad9bc71a16177a9a2b03379d2948"
server_key_iv: "0e0484e20187681c565c91430bb1dc35"
server_key_version: "encryptionkey@0"
I20260923 21:05:41.797314 20368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3934f642b43c4a18b6e60d6563f1d902"
format_stamp: "Formatted at 2026-09-23 21:05:41 on dist-test-slave-8xn5"
server_key: "bbd1ad9bc71a16177a9a2b03379d2948"
server_key_iv: "0e0484e20187681c565c91430bb1dc35"
server_key_version: "encryptionkey@0"
I20260923 21:05:41.803450 20368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 21:05:41.807783 20382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:41.809501 20368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:05:41.809721 20368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/wal
uuid: "3934f642b43c4a18b6e60d6563f1d902"
format_stamp: "Formatted at 2026-09-23 21:05:41 on dist-test-slave-8xn5"
server_key: "bbd1ad9bc71a16177a9a2b03379d2948"
server_key_iv: "0e0484e20187681c565c91430bb1dc35"
server_key_version: "encryptionkey@0"
I20260923 21:05:41.809980 20368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:41.829351 20368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:41.830420 20368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:41.830744 20368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:41.831838 20368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:41.833628 20368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:41.833770 20368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:41.833923 20368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:41.834029 20368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:41.870074 20368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42113
I20260923 21:05:41.870155 20494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42113 every 8 connection(s)
I20260923 21:05:41.871579 20368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/data/info.pb
I20260923 21:05:41.876581 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 20368
I20260923 21:05:41.876786 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-0/wal/instance
I20260923 21:05:41.877275 388 external_mini_cluster.cc:1428] Setting key 91fb87b1ed303c3d50b001291db70362
I20260923 21:05:41.880230 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39041
--builtin_ntp_servers=127.0.97.20:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 21:05:41.885550 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:41.874480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42117 (local address 127.0.97.62:39041)
0923 21:05:41.874914 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:41.874918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:41.876144 (+ 1226us) server_negotiation.cc:407] Connection header received
0923 21:05:41.878224 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:41.878230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:41.878355 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:41.878543 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:41.879910 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:41.880762 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:41.882472 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:41.882905 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:41.883051 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:41.884530 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:41.884535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:41.884538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:41.884580 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:41.884636 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:41.884653 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:41.884776 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:41.885058 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 21:05:41.885163 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":240,"threads_started":1}
I20260923 21:05:41.886878 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:41.875020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39041 (local address 127.0.97.1:42117)
0923 21:05:41.875942 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:41.875980 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:41.877875 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:41.878797 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:41.878808 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:41.878905 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:41.879734 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:41.879754 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:41.880944 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:41.880948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:41.882247 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:41.882259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:41.883152 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:41.883159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:41.883401 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:41.884154 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:41.884158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:41.884201 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:41.884289 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:41.884776 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:41.884791 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:41.884802 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 21:05:41.885048 (+ 246us) client_negotiation.cc:240] Negotiation successful
0923 21:05:41.885491 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":290,"threads_started":1}
I20260923 21:05:41.888866 20495 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39041
I20260923 21:05:41.889249 20495 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:41.890259 20495 heartbeater.cc:511] Master 127.0.97.62:39041 requested a full tablet report, sending...
I20260923 21:05:41.892735 20309 ts_manager.cc:194] Registered new tserver with Master: 3934f642b43c4a18b6e60d6563f1d902 (127.0.97.1:42113)
I20260923 21:05:41.894739 20309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:42117
W20260923 21:05:42.234547 20499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:42.234848 20499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:42.234995 20499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:42.243141 20499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:42.243301 20499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:42.243500 20499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260923 21:05:42.253973 20499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:42.255432 20499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:42.257493 20499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:42.272989 20504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:42.272945 20505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:42.272882 20507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:42.274113 20499 server_base.cc:1034] running on GCE node
I20260923 21:05:42.275125 20499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:42.276674 20499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:42.278092 20499 hybrid_clock.cc:648] HybridClock initialized: now 1790197542277965 us; error 87 us; skew 500 ppm
I20260923 21:05:42.278620 20499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:42.283257 20499 webserver.cc:460] Webserver started at http://127.0.97.2:41643/ using document root <none> and password file <none>
I20260923 21:05:42.284273 20499 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:42.284377 20499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:42.284765 20499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:42.289477 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data/instance:
uuid: "0efd68545a1846d6986a299d7333c316"
format_stamp: "Formatted at 2026-09-23 21:05:42 on dist-test-slave-8xn5"
server_key: "92c2f70d141d15b051bf128b7042da20"
server_key_iv: "cc8928e8fb3ed1ea86edc3fe3c8c1b8f"
server_key_version: "encryptionkey@0"
I20260923 21:05:42.290858 20499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0efd68545a1846d6986a299d7333c316"
format_stamp: "Formatted at 2026-09-23 21:05:42 on dist-test-slave-8xn5"
server_key: "92c2f70d141d15b051bf128b7042da20"
server_key_iv: "cc8928e8fb3ed1ea86edc3fe3c8c1b8f"
server_key_version: "encryptionkey@0"
I20260923 21:05:42.298861 20499 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260923 21:05:42.304766 20513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:42.307113 20499 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260923 21:05:42.307330 20499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal
uuid: "0efd68545a1846d6986a299d7333c316"
format_stamp: "Formatted at 2026-09-23 21:05:42 on dist-test-slave-8xn5"
server_key: "92c2f70d141d15b051bf128b7042da20"
server_key_iv: "cc8928e8fb3ed1ea86edc3fe3c8c1b8f"
server_key_version: "encryptionkey@0"
I20260923 21:05:42.307636 20499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:42.341867 20499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:42.342940 20499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:42.343266 20499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:42.344375 20499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:42.346086 20499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:42.346184 20499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:42.346364 20499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:42.346444 20499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260923 21:05:42.383499 20499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40843
I20260923 21:05:42.383634 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40843 every 8 connection(s)
I20260923 21:05:42.385162 20499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/data/info.pb
I20260923 21:05:42.390062 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 20499
I20260923 21:05:42.390249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal/instance
I20260923 21:05:42.390935 388 external_mini_cluster.cc:1428] Setting key b8e8dd273e373f9a7b9538a15a68f00a
I20260923 21:05:42.394390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39041
--builtin_ntp_servers=127.0.97.20:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 21:05:42.402566 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.388605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48351 (local address 127.0.97.62:39041)
0923 21:05:42.389208 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:42.389215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:42.391458 (+ 2243us) server_negotiation.cc:407] Connection header received
0923 21:05:42.393839 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:42.393844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:42.393912 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:42.394032 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:42.395646 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.396591 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.399224 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.399666 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.399773 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.401600 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:42.401605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:42.401608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:42.401638 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:42.401681 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:42.401694 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:42.401762 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:42.402125 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 21:05:42.402216 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":278,"threads_started":1}
I20260923 21:05:42.403983 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.389399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39041 (local address 127.0.97.2:48351)
0923 21:05:42.391241 (+ 1842us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:42.391307 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:42.393495 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:42.394381 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:42.394394 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:42.394541 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:42.395436 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.395464 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.396774 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.396782 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.399006 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.399031 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.399922 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.399933 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.400318 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.401175 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:42.401181 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:42.401239 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:42.401363 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:42.401795 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:42.401808 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:42.401819 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 21:05:42.402014 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 21:05:42.402537 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"thread_start_us":949,"threads_started":1}
I20260923 21:05:42.405663 20626 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39041
I20260923 21:05:42.405997 20626 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:42.407224 20626 heartbeater.cc:511] Master 127.0.97.62:39041 requested a full tablet report, sending...
I20260923 21:05:42.408963 20309 ts_manager.cc:194] Registered new tserver with Master: 0efd68545a1846d6986a299d7333c316 (127.0.97.2:40843)
I20260923 21:05:42.409878 20309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:48351
W20260923 21:05:42.750826 20630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:42.751184 20630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:42.751353 20630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:42.759089 20630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:42.759243 20630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:42.759425 20630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260923 21:05:42.768940 20630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:42.770380 20630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:42.772310 20630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:42.786659 20635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:42.786911 20638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:42.786685 20636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:42.788120 20630 server_base.cc:1034] running on GCE node
I20260923 21:05:42.788897 20630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:42.790056 20630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:42.791306 20630 hybrid_clock.cc:648] HybridClock initialized: now 1790197542791241 us; error 56 us; skew 500 ppm
I20260923 21:05:42.791683 20630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 21:05:42.794410 20630 webserver.cc:460] Webserver started at http://127.0.97.3:37777/ using document root <none> and password file <none>
I20260923 21:05:42.795171 20630 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:42.795310 20630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:42.795660 20630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:42.798880 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data/instance:
uuid: "b67da395ab9b43f3a2ab6a01d9c5c492"
format_stamp: "Formatted at 2026-09-23 21:05:42 on dist-test-slave-8xn5"
server_key: "fbdc7a6912cbf36f10f8526ec7acbabe"
server_key_iv: "dc51536f1fbbcd61af3978400ac94799"
server_key_version: "encryptionkey@0"
I20260923 21:05:42.799818 20630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b67da395ab9b43f3a2ab6a01d9c5c492"
format_stamp: "Formatted at 2026-09-23 21:05:42 on dist-test-slave-8xn5"
server_key: "fbdc7a6912cbf36f10f8526ec7acbabe"
server_key_iv: "dc51536f1fbbcd61af3978400ac94799"
server_key_version: "encryptionkey@0"
I20260923 21:05:42.805511 20630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:05:42.809630 20644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:42.811420 20630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 21:05:42.811664 20630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal
uuid: "b67da395ab9b43f3a2ab6a01d9c5c492"
format_stamp: "Formatted at 2026-09-23 21:05:42 on dist-test-slave-8xn5"
server_key: "fbdc7a6912cbf36f10f8526ec7acbabe"
server_key_iv: "dc51536f1fbbcd61af3978400ac94799"
server_key_version: "encryptionkey@0"
I20260923 21:05:42.811913 20630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:42.843595 20630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:42.844605 20630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:42.844904 20630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:42.845988 20630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 21:05:42.847616 20630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 21:05:42.847745 20630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:42.847888 20630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 21:05:42.847978 20630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:42.883813 20630 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37043
I20260923 21:05:42.884054 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37043 every 8 connection(s)
I20260923 21:05:42.885281 20630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/data/info.pb
I20260923 21:05:42.895560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 20630
I20260923 21:05:42.895741 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal/instance
I20260923 21:05:42.896267 388 external_mini_cluster.cc:1428] Setting key d1f6504338e1d9453ad27844ed869094
I20260923 21:05:42.898880 20495 heartbeater.cc:503] Master 127.0.97.62:39041 was elected leader, sending a full tablet report...
I20260923 21:05:42.899219 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.888107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55205 (local address 127.0.97.62:39041)
0923 21:05:42.888317 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:42.888326 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:42.890002 (+ 1676us) server_negotiation.cc:407] Connection header received
0923 21:05:42.892028 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:42.892033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:42.892141 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:42.892261 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:42.893369 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.894138 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.896171 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.896622 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.896772 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.898242 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:42.898247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:42.898250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:42.898269 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:42.898302 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:42.898314 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:42.898475 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:42.898781 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 21:05:42.898884 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:05:42.901113 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.888723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39041 (local address 127.0.97.3:55205)
0923 21:05:42.889818 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:42.889883 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:42.891563 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:42.892472 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:42.892481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:42.892575 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:05:42.893182 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.893199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.894273 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.894277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.895966 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.895981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.896888 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.896896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.897121 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.897882 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 21:05:42.897886 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:05:42.897938 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 21:05:42.898021 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:05:42.898496 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:05:42.898508 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:05:42.898516 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 21:05:42.899400 (+ 884us) client_negotiation.cc:240] Negotiation successful
0923 21:05:42.899837 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":326,"threads_started":1}
I20260923 21:05:42.902627 20757 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39041
I20260923 21:05:42.903024 20757 heartbeater.cc:464] Registering TS with master...
I20260923 21:05:42.903884 20757 heartbeater.cc:511] Master 127.0.97.62:39041 requested a full tablet report, sending...
I20260923 21:05:42.905339 20309 ts_manager.cc:194] Registered new tserver with Master: b67da395ab9b43f3a2ab6a01d9c5c492 (127.0.97.3:37043)
I20260923 21:05:42.906225 20309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:55205
I20260923 21:05:42.912143 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 21:05:42.923358 20628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.916820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.97.62:39041)
0923 21:05:42.917030 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:42.917038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:42.917568 (+ 530us) server_negotiation.cc:407] Connection header received
0923 21:05:42.917788 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:42.917792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:42.917860 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:42.917975 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:42.918530 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.919312 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.920723 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.921152 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.921293 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.922549 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:42.922554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:42.922558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:42.922583 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:42.922620 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:42.922634 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:42.922738 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:42.922901 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 21:05:42.923001 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 21:05:42.928539 20309 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 21:05:42.935688 20309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 21:05:42.961086 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.949720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.0.97.1:42113)
0923 21:05:42.951386 (+ 1666us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:42.951393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:42.951426 (+ 33us) server_negotiation.cc:407] Connection header received
0923 21:05:42.951536 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:42.951544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:42.951826 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:42.952769 (+ 943us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:42.953060 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.954750 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.957898 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.959683 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.959845 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.960169 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:42.960450 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 21:05:42.960649 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492,"thread_start_us":1355,"threads_started":1}
I20260923 21:05:42.961114 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.948943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45454)
0923 21:05:42.949821 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:42.949880 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:42.950117 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:42.952074 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:42.952081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:42.952117 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:42.952553 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.952567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.954919 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.954926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.957590 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.957606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.959912 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.959919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.960149 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.960218 (+ 69us) client_negotiation.cc:769] Sending connection context
0923 21:05:42.960392 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 21:05:42.960635 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":28,"thread_start_us":995,"threads_started":1}
I20260923 21:05:42.961452 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.949126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.0.97.2:40843)
0923 21:05:42.950935 (+ 1809us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:42.950947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:42.951834 (+ 887us) server_negotiation.cc:407] Connection header received
0923 21:05:42.952123 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:42.952130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:42.952354 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:42.952543 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:42.953522 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.955057 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.958496 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.959973 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.960108 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.960335 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:42.960889 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 21:05:42.961071 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1644,"thread_start_us":1538,"threads_started":1}
I20260923 21:05:42.962286 20770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.949003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40843 (local address 127.0.0.1:46940)
0923 21:05:42.951722 (+ 2719us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:42.951739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:42.951940 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:42.952849 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:42.952857 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:42.952883 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:42.953296 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.953307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.955240 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.955245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.958250 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.958264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.960469 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.960475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.960673 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.960708 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 21:05:42.961718 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0923 21:05:42.961884 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2607,"mutex_wait_us":1,"thread_start_us":2454,"threads_started":1}
I20260923 21:05:42.966154 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.952134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37043 (local address 127.0.0.1:35776)
0923 21:05:42.957236 (+ 5102us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:42.957254 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:42.957428 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:42.958784 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:42.958790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:42.958820 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:42.959246 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.959258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.961209 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.961214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.963384 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:42.963403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.965349 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.965356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:42.965552 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.965588 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 21:05:42.965680 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 21:05:42.965823 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4958,"thread_start_us":4782,"threads_started":1}
I20260923 21:05:42.966358 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:42.952258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.0.97.3:37043)
0923 21:05:42.956751 (+ 4493us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:42.956760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:42.957372 (+ 612us) server_negotiation.cc:407] Connection header received
0923 21:05:42.957621 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:42.957632 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:42.957929 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:42.958156 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:42.959513 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.961033 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.963651 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:42.965109 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:42.965273 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:42.965565 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:42.965767 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 21:05:42.965958 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4275,"thread_start_us":4139,"threads_started":1}
I20260923 21:05:42.966135 20430 tablet_service.cc:1467] Processing CreateTablet for tablet 123be5a9e1ef434083e9a1289c8e8e4d (DEFAULT_TABLE table=test-table [id=28a29c801263490da03edfc2ed8778b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:42.966132 20561 tablet_service.cc:1467] Processing CreateTablet for tablet 123be5a9e1ef434083e9a1289c8e8e4d (DEFAULT_TABLE table=test-table [id=28a29c801263490da03edfc2ed8778b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:42.967679 20561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123be5a9e1ef434083e9a1289c8e8e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:42.967711 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123be5a9e1ef434083e9a1289c8e8e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:42.970471 20692 tablet_service.cc:1467] Processing CreateTablet for tablet 123be5a9e1ef434083e9a1289c8e8e4d (DEFAULT_TABLE table=test-table [id=28a29c801263490da03edfc2ed8778b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 21:05:42.972070 20692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123be5a9e1ef434083e9a1289c8e8e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:42.979385 20775 tablet_bootstrap.cc:492] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: Bootstrap starting.
I20260923 21:05:42.983484 20776 tablet_bootstrap.cc:492] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Bootstrap starting.
I20260923 21:05:42.984198 20775 tablet_bootstrap.cc:654] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:42.985584 20775 log.cc:824] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:42.987267 20776 tablet_bootstrap.cc:654] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:42.988067 20778 tablet_bootstrap.cc:492] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Bootstrap starting.
I20260923 21:05:42.988731 20776 log.cc:824] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:42.989161 20775 tablet_bootstrap.cc:492] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: No bootstrap required, opened a new log
I20260923 21:05:42.989503 20775 ts_tablet_manager.cc:1397] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 21:05:42.992045 20778 tablet_bootstrap.cc:654] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:42.992070 20776 tablet_bootstrap.cc:492] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: No bootstrap required, opened a new log
I20260923 21:05:42.992331 20776 ts_tablet_manager.cc:1397] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 21:05:42.993177 20778 log.cc:824] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:42.996200 20778 tablet_bootstrap.cc:492] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: No bootstrap required, opened a new log
I20260923 21:05:42.996479 20778 ts_tablet_manager.cc:1397] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 21:05:43.000602 20775 raft_consensus.cc:348] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.001101 20775 raft_consensus.cc:374] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:43.001199 20775 raft_consensus.cc:729] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3934f642b43c4a18b6e60d6563f1d902, State: Initialized, Role: FOLLOWER
I20260923 21:05:43.001999 20775 consensus_queue.cc:260] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.003209 20776 raft_consensus.cc:348] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.003795 20776 raft_consensus.cc:374] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:43.003929 20776 raft_consensus.cc:729] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0efd68545a1846d6986a299d7333c316, State: Initialized, Role: FOLLOWER
I20260923 21:05:43.004737 20776 consensus_queue.cc:260] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.004853 20778 raft_consensus.cc:348] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.005208 20778 raft_consensus.cc:374] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:43.005336 20778 raft_consensus.cc:729] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67da395ab9b43f3a2ab6a01d9c5c492, State: Initialized, Role: FOLLOWER
I20260923 21:05:43.006134 20626 heartbeater.cc:503] Master 127.0.97.62:39041 was elected leader, sending a full tablet report...
I20260923 21:05:43.006428 20778 consensus_queue.cc:260] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.006909 20776 ts_tablet_manager.cc:1428] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260923 21:05:43.009074 20775 ts_tablet_manager.cc:1428] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260923 21:05:43.010159 20757 heartbeater.cc:503] Master 127.0.97.62:39041 was elected leader, sending a full tablet report...
I20260923 21:05:43.010848 20778 ts_tablet_manager.cc:1428] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260923 21:05:43.124469 20496 tablet.cc:2369] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:43.137223 20758 tablet.cc:2369] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 21:05:43.137954 20627 tablet.cc:2369] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:05:43.197327 20783 raft_consensus.cc:482] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 21:05:43.197790 20783 raft_consensus.cc:504] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.199857 20783 leader_election.cc:288] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0efd68545a1846d6986a299d7333c316 (127.0.97.2:40843), 3934f642b43c4a18b6e60d6563f1d902 (127.0.97.1:42113)
I20260923 21:05:43.206594 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.200445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45625 (local address 127.0.97.1:42113)
0923 21:05:43.200665 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:43.200674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:43.200898 (+ 224us) server_negotiation.cc:407] Connection header received
0923 21:05:43.201149 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:43.201154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:43.201231 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:43.201353 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:43.201852 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.202827 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.204699 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.205569 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.205725 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.205829 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:43.206135 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 21:05:43.206282 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:05:43.206605 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.200445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.97.3:45625)
0923 21:05:43.200779 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.200796 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:43.200979 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:43.201432 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:43.201439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:43.201469 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:43.201685 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:43.201695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.202963 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.202971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:43.204516 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:43.204532 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.205754 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.205760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:43.205948 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.205974 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 21:05:43.206104 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 21:05:43.206268 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":3}
I20260923 21:05:43.207547 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.200467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57689 (local address 127.0.97.2:40843)
0923 21:05:43.200689 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:43.200695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:43.201663 (+ 968us) server_negotiation.cc:407] Connection header received
0923 21:05:43.201907 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:43.201913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:43.201999 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:43.202106 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 21:05:43.202798 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.203556 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.205418 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.206572 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.206718 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.206813 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:43.207137 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 21:05:43.207269 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 21:05:43.207592 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.200551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40843 (local address 127.0.97.3:57689)
0923 21:05:43.201537 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.201557 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:05:43.201731 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:05:43.202126 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:05:43.202131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:05:43.202156 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 21:05:43.202626 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:43.202639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.203723 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.203729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:43.205221 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:05:43.205233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.206768 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.206775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:05:43.206964 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.206988 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 21:05:43.207096 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 21:05:43.207236 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":588,"threads_started":1}
I20260923 21:05:43.208077 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123be5a9e1ef434083e9a1289c8e8e4d" candidate_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934f642b43c4a18b6e60d6563f1d902" is_pre_election: true
I20260923 21:05:43.208786 20450 raft_consensus.cc:2454] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67da395ab9b43f3a2ab6a01d9c5c492 in term 0.
I20260923 21:05:43.208766 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123be5a9e1ef434083e9a1289c8e8e4d" candidate_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efd68545a1846d6986a299d7333c316" is_pre_election: true
I20260923 21:05:43.209404 20581 raft_consensus.cc:2454] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b67da395ab9b43f3a2ab6a01d9c5c492 in term 0.
I20260923 21:05:43.209746 20647 leader_election.cc:302] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3934f642b43c4a18b6e60d6563f1d902, b67da395ab9b43f3a2ab6a01d9c5c492; no voters:
I20260923 21:05:43.210242 20783 raft_consensus.cc:2781] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 21:05:43.210435 20783 raft_consensus.cc:482] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 21:05:43.210554 20783 raft_consensus.cc:3037] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:43.213018 20783 raft_consensus.cc:504] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.213804 20783 leader_election.cc:288] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [CANDIDATE]: Term 1 election: Requested vote from peers 0efd68545a1846d6986a299d7333c316 (127.0.97.2:40843), 3934f642b43c4a18b6e60d6563f1d902 (127.0.97.1:42113)
I20260923 21:05:43.214371 20581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123be5a9e1ef434083e9a1289c8e8e4d" candidate_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0efd68545a1846d6986a299d7333c316"
I20260923 21:05:43.214480 20450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123be5a9e1ef434083e9a1289c8e8e4d" candidate_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3934f642b43c4a18b6e60d6563f1d902"
I20260923 21:05:43.214602 20581 raft_consensus.cc:3037] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:43.214761 20450 raft_consensus.cc:3037] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:43.217353 20581 raft_consensus.cc:2454] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67da395ab9b43f3a2ab6a01d9c5c492 in term 1.
I20260923 21:05:43.217582 20450 raft_consensus.cc:2454] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b67da395ab9b43f3a2ab6a01d9c5c492 in term 1.
I20260923 21:05:43.217840 20645 leader_election.cc:302] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0efd68545a1846d6986a299d7333c316, b67da395ab9b43f3a2ab6a01d9c5c492; no voters:
I20260923 21:05:43.218206 20783 raft_consensus.cc:2781] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:43.218779 20783 raft_consensus.cc:686] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 1 LEADER]: Becoming Leader. State: Replica: b67da395ab9b43f3a2ab6a01d9c5c492, State: Running, Role: LEADER
I20260923 21:05:43.219332 20783 consensus_queue.cc:237] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } }
I20260923 21:05:43.224334 20307 catalog_manager.cc:5495] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 reported cstate change: term changed from 0 to 1, leader changed from <none> to b67da395ab9b43f3a2ab6a01d9c5c492 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b67da395ab9b43f3a2ab6a01d9c5c492" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37043 } health_report { overall_health: HEALTHY } } }
I20260923 21:05:43.261590 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.255248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.0.97.3:37043)
0923 21:05:43.255546 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:43.255552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:43.255573 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:05:43.255818 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:43.255822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:43.255935 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:43.256140 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:05:43.256554 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.257412 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.259213 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.259674 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.259843 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.260404 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:05:43.260946 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:05:43.261136 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:43.261200 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 21:05:43.261292 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 21:05:43.271102 20581 raft_consensus.cc:1261] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 1 FOLLOWER]: Refusing update from remote peer b67da395ab9b43f3a2ab6a01d9c5c492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:43.271099 20450 raft_consensus.cc:1261] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 1 FOLLOWER]: Refusing update from remote peer b67da395ab9b43f3a2ab6a01d9c5c492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:05:43.272145 20783 consensus_queue.cc:1035] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0efd68545a1846d6986a299d7333c316" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:43.272503 20785 consensus_queue.cc:1035] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3934f642b43c4a18b6e60d6563f1d902" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:05:43.285256 20788 mvcc.cc:205] Tried to move back new op lower bound from 7332649137226788864 to 7332649137030946816. Current Snapshot: MvccSnapshot[applied={T|T < 7332649137226788864}]
I20260923 21:05:43.301792 20307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:43.302096 20307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 21:05:43.306523 20307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P def08aac7d2f400ab4468c65caecb980: Sending DeleteTablet for 3 replicas of tablet 123be5a9e1ef434083e9a1289c8e8e4d
I20260923 21:05:43.308063 20430 tablet_service.cc:1514] Processing DeleteTablet for tablet 123be5a9e1ef434083e9a1289c8e8e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:43 UTC) from {username='slave'} at 127.0.0.1:45454
I20260923 21:05:43.308264 20561 tablet_service.cc:1514] Processing DeleteTablet for tablet 123be5a9e1ef434083e9a1289c8e8e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:43 UTC) from {username='slave'} at 127.0.0.1:46940
I20260923 21:05:43.308274 20692 tablet_service.cc:1514] Processing DeleteTablet for tablet 123be5a9e1ef434083e9a1289c8e8e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 21:05:43 UTC) from {username='slave'} at 127.0.0.1:35776
I20260923 21:05:43.313306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20368
I20260923 21:05:43.333752 20800 tablet_replica.cc:330] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: stopping tablet replica
I20260923 21:05:43.334458 20800 raft_consensus.cc:2229] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 1 LEADER]: Raft consensus shutting down.
I20260923 21:05:43.335438 20800 raft_consensus.cc:2258] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:43.344956 20800 ts_tablet_manager.cc:1905] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:43.347059 20798 tablet_replica.cc:330] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: stopping tablet replica
I20260923 21:05:43.347715 20798 raft_consensus.cc:2229] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:43.348438 20798 raft_consensus.cc:2258] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:43.351348 20798 ts_tablet_manager.cc:1905] T 123be5a9e1ef434083e9a1289c8e8e4d P 3934f642b43c4a18b6e60d6563f1d902: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:43.351981 20799 tablet_replica.cc:330] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: stopping tablet replica
I20260923 21:05:43.352442 20799 raft_consensus.cc:2229] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 21:05:43.352962 20799 raft_consensus.cc:2258] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 21:05:43.355711 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.314859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.0.97.1:42113)
0923 21:05:43.315075 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:43.315083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:43.332438 (+ 17355us) server_negotiation.cc:407] Connection header received
0923 21:05:43.336040 (+ 3602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:43.336051 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:43.336220 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:43.336327 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:43.337362 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.339013 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.348994 (+ 9981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.350065 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.350228 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.353805 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:43.353818 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:43.353836 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:43.354671 (+ 835us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:43.354748 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:43.354778 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:43.354889 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:43.355173 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 21:05:43.355298 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 21:05:43.357002 20490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:43.359493 20799 ts_tablet_manager.cc:1905] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 21:05:43.363610 20800 ts_tablet_manager.cc:1918] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:43.363817 20800 log.cc:1196] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-2/wal/wals/123be5a9e1ef434083e9a1289c8e8e4d
I20260923 21:05:43.364401 20800 ts_tablet_manager.cc:1939] T 123be5a9e1ef434083e9a1289c8e8e4d P b67da395ab9b43f3a2ab6a01d9c5c492: Deleting consensus metadata
I20260923 21:05:43.366276 20293 catalog_manager.cc:4841] TS b67da395ab9b43f3a2ab6a01d9c5c492 (127.0.97.3:37043): tablet 123be5a9e1ef434083e9a1289c8e8e4d (table test-table [id=28a29c801263490da03edfc2ed8778b9]) successfully deleted
I20260923 21:05:43.368463 20799 ts_tablet_manager.cc:1918] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 21:05:43.368690 20799 log.cc:1196] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Deleting WAL directory at /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790197207951953-388-0/minicluster-data/ts-1/wal/wals/123be5a9e1ef434083e9a1289c8e8e4d
I20260923 21:05:43.369352 20799 ts_tablet_manager.cc:1939] T 123be5a9e1ef434083e9a1289c8e8e4d P 0efd68545a1846d6986a299d7333c316: Deleting consensus metadata
I20260923 21:05:43.371845 20293 catalog_manager.cc:4841] TS 0efd68545a1846d6986a299d7333c316 (127.0.97.2:40843): tablet 123be5a9e1ef434083e9a1289c8e8e4d (table test-table [id=28a29c801263490da03edfc2ed8778b9]) successfully deleted
I20260923 21:05:43.471195 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 20368
W20260923 21:05:43.496598 20295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42113 (error 108)
I20260923 21:05:43.497408 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20499
I20260923 21:05:43.499924 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.498266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45478)
0923 21:05:43.499460 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.499667 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":928,"threads_started":1}
W20260923 21:05:43.500356 20295 catalog_manager.cc:4584] TS 3934f642b43c4a18b6e60d6563f1d902 (127.0.97.1:42113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 123be5a9e1ef434083e9a1289c8e8e4d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
I20260923 21:05:43.509689 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.499884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.0.97.2:40843)
0923 21:05:43.500110 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:43.500117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:43.500142 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:05:43.500225 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:43.500231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:43.500356 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:43.500523 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:43.501241 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.502748 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.505434 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.506023 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.506202 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.507970 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:43.507981 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:43.507999 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:43.508745 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:43.508803 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:43.508836 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:43.509020 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:43.509146 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 21:05:43.509277 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 21:05:43.510521 20621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:43.555080 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.554585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45490)
0923 21:05:43.554818 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.554916 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 21:05:43.556351 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.555961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45494)
0923 21:05:43.556128 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.556211 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 21:05:43.617509 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.616810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45500)
0923 21:05:43.617095 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.617261 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 21:05:43.618755 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.618312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45504)
0923 21:05:43.618530 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.618613 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 21:05:43.651757 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 20499
I20260923 21:05:43.675573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20630
I20260923 21:05:43.684218 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.676488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.0.97.3:37043)
0923 21:05:43.676675 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:43.676679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:43.676782 (+ 103us) server_negotiation.cc:407] Connection header received
0923 21:05:43.676926 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:43.676930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:43.677037 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:43.677132 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:43.677630 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.678565 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.680445 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:43.680946 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:43.681051 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:43.682436 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:43.682447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:43.682468 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:43.683317 (+ 849us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:43.683396 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:43.683431 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:43.683549 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:43.683677 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 21:05:43.683797 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:05:43.685099 20752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:43.699622 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.699094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45512)
0923 21:05:43.699357 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.699456 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 21:05:43.701023 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.700448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45520)
0923 21:05:43.700683 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.700784 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260923 21:05:43.810788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 20630
I20260923 21:05:43.830614 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20278
I20260923 21:05:43.831813 20339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:43.945530 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:43.944931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42113 (local address 127.0.0.1:45534)
0923 21:05:43.945186 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 21:05:43.945346 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 21:05:43.945689 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 20278
2026-09-23T21:05:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3250 ms)
[----------] 6 tests from EncryptionPolicyTest (23538 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:44 dist-test-slave-8xn5 krb5kdc[20810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:44 dist-test-slave-8xn5 krb5kdc[20810](info): set up 2 sockets
Sep 23 21:05:44 dist-test-slave-8xn5 krb5kdc[20810](info): commencing operation
krb5kdc: starting...
W20260923 21:05:45.098902 388 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:45 dist-test-slave-8xn5 krb5kdc[20810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:45.375114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:45.720077 20823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:45.720378 20823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:45.720552 20823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:45.727864 20823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:45.727988 20823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:45.728035 20823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:45.728155 20823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:45.728219 20823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:45.739209 20823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40253
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:45.740612 20823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:45.742046 20823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:45.754446 20828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:45.754544 20829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:45.754513 20831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:45.755604 20823 server_base.cc:1034] running on GCE node
I20260923 21:05:45.756479 20823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:45.758183 20823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:45.759445 20823 hybrid_clock.cc:648] HybridClock initialized: now 1790197545759387 us; error 73 us; skew 500 ppm
I20260923 21:05:45.759918 20823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:45 dist-test-slave-8xn5 krb5kdc[20810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:45.763336 20823 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:05:45.764882 20823 webserver.cc:460] Webserver started at http://127.0.0.1:34315/ using document root <none> and password file <none>
I20260923 21:05:45.765643 20823 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:45.765836 20823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:45.766104 20823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:45.769263 20823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "8908f430199c4fb8b42258f19b7b7058"
format_stamp: "Formatted at 2026-09-23 21:05:45 on dist-test-slave-8xn5"
server_key: "8b81d7ab8d45e7bf579b68a11834cd42"
server_key_iv: "8527cb25fe0bf29821956179e31d1007"
server_key_version: "encryptionkey@0"
I20260923 21:05:45.770083 20823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "8908f430199c4fb8b42258f19b7b7058"
format_stamp: "Formatted at 2026-09-23 21:05:45 on dist-test-slave-8xn5"
server_key: "8b81d7ab8d45e7bf579b68a11834cd42"
server_key_iv: "8527cb25fe0bf29821956179e31d1007"
server_key_version: "encryptionkey@0"
I20260923 21:05:45.775974 20823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 21:05:45.780007 20838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:45.781757 20823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 21:05:45.782002 20823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "8908f430199c4fb8b42258f19b7b7058"
format_stamp: "Formatted at 2026-09-23 21:05:45 on dist-test-slave-8xn5"
server_key: "8b81d7ab8d45e7bf579b68a11834cd42"
server_key_iv: "8527cb25fe0bf29821956179e31d1007"
server_key_version: "encryptionkey@0"
I20260923 21:05:45.782229 20823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:45.799490 20823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:45.804210 20823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:45.804538 20823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:45.824242 20823 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40253
I20260923 21:05:45.824275 20889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40253 every 8 connection(s)
I20260923 21:05:45.825703 20823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:45.829752 20890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:45.831681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 20823
I20260923 21:05:45.831900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:45.832603 388 external_mini_cluster.cc:1428] Setting key a1abfd81a76fcd957db1428b321ee768
I20260923 21:05:45.841676 20890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058: Bootstrap starting.
I20260923 21:05:45.845057 20890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:45.846155 20890 log.cc:824] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:05:45 dist-test-slave-8xn5 krb5kdc[20810](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 21:05:45.849431 20890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058: No bootstrap required, opened a new log
I20260923 21:05:45.857687 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:45.834717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41493 (local address 127.0.0.1:40253)
0923 21:05:45.835622 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:45.835634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:45.835680 (+ 46us) server_negotiation.cc:407] Connection header received
0923 21:05:45.837401 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:45.837425 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:45.837843 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:45.838249 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:45.839795 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:45.841102 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:45.842627 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:45.843332 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:45.843555 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:45.848458 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:45.848468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:45.848484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:45.848550 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:45.852343 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:45.853480 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:45.853488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:45.853495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:45.853690 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:45.854205 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:45.854209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:45.854211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:45.855066 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:45.855376 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:45.856001 (+ 625us) server_negotiation.cc:299] Negotiation successful
0923 21:05:45.856461 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":424,"threads_started":1}
I20260923 21:05:45.858817 20890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8908f430199c4fb8b42258f19b7b7058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40253 } }
I20260923 21:05:45.859267 20890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:45.859411 20890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8908f430199c4fb8b42258f19b7b7058, State: Initialized, Role: FOLLOWER
I20260923 21:05:45.860072 20890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8908f430199c4fb8b42258f19b7b7058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40253 } }
I20260923 21:05:45.860337 20890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:45.860476 20890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:45.860687 20890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:45.862588 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:05:45.863725 20890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8908f430199c4fb8b42258f19b7b7058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40253 } }
I20260923 21:05:45.864342 20890 leader_election.cc:302] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8908f430199c4fb8b42258f19b7b7058; no voters:
I20260923 21:05:45.865031 20890 leader_election.cc:288] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:45.865168 20895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:45.865908 20895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [term 1 LEADER]: Becoming Leader. State: Replica: 8908f430199c4fb8b42258f19b7b7058, State: Running, Role: LEADER
I20260923 21:05:45.866657 20895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8908f430199c4fb8b42258f19b7b7058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40253 } }
I20260923 21:05:45.868690 20895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8908f430199c4fb8b42258f19b7b7058. Latest consensus state: current_term: 1 leader_uuid: "8908f430199c4fb8b42258f19b7b7058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8908f430199c4fb8b42258f19b7b7058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40253 } } }
I20260923 21:05:45.869082 20895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:45.869596 20897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8908f430199c4fb8b42258f19b7b7058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8908f430199c4fb8b42258f19b7b7058" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40253 } } }
I20260923 21:05:45.870072 20897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:45.879038 20890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:45.879287 20901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:45.885286 20901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:45.902565 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:45.878505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60763 (local address 127.0.0.1:40253)
0923 21:05:45.878745 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:45.878756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:45.879219 (+ 463us) server_negotiation.cc:407] Connection header received
0923 21:05:45.879569 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:45.879576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:45.879666 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:45.879788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:45.882243 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:45.883952 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:45.886696 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:45.887632 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:45.887773 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:45.891467 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:45.891477 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:45.891485 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:45.891549 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:45.896644 (+ 5095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:45.898174 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:45.898183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:45.898190 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:45.898469 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:45.899230 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:45.899238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:45.899244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:45.899811 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:45.900183 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:45.901711 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0923 21:05:45.901991 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:05:45.909040 20901 catalog_manager.cc:1347] Generated new cluster ID: 15fedc56978945b598a5a202c3a4e2e4
I20260923 21:05:45.909159 20901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:45.919281 20901 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:45.920621 20901 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:45.931491 20901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8908f430199c4fb8b42258f19b7b7058: Generated new TSK 0
I20260923 21:05:45.932767 20901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:45.946939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20823
I20260923 21:05:45.948733 20885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:46.043079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 20823
W20260923 21:05:46.061087 20808 connection.cc:535] client connection to 127.0.0.1:40253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40253): Connection reset by peer (error 104)
2026-09-23T21:05:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:46 dist-test-slave-8xn5 krb5kdc[20922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:46 dist-test-slave-8xn5 krb5kdc[20922](info): set up 2 sockets
Sep 23 21:05:46 dist-test-slave-8xn5 krb5kdc[20922](info): commencing operation
krb5kdc: starting...
W20260923 21:05:47.243167 388 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:47 dist-test-slave-8xn5 krb5kdc[20922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:47.522882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:47.867851 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:47.868170 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:47.868325 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:47.875981 20935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:47.876106 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:47.876214 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:47.876338 20935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:47.876427 20935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:47.886742 20935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42555
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:47.888134 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:47.889525 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:47.902590 20940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:47.902540 20943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:47.902633 20941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:47.903563 20935 server_base.cc:1034] running on GCE node
I20260923 21:05:47.904397 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:47.906121 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:47.907384 20935 hybrid_clock.cc:648] HybridClock initialized: now 1790197547907322 us; error 66 us; skew 500 ppm
I20260923 21:05:47.907768 20935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:47 dist-test-slave-8xn5 krb5kdc[20922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:47.915171 20935 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:05:47.917028 20935 webserver.cc:460] Webserver started at http://127.0.0.1:43857/ using document root <none> and password file <none>
I20260923 21:05:47.917810 20935 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:47.917965 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:47.918298 20935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:47.921551 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "0f19c083c853480983aeead9b4e49262"
format_stamp: "Formatted at 2026-09-23 21:05:47 on dist-test-slave-8xn5"
server_key: "3957ee10afcd6e135943bb360fe5b1fa"
server_key_iv: "17084d59306d1b53f276dbeff3b7d435"
server_key_version: "encryptionkey@0"
I20260923 21:05:47.922456 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f19c083c853480983aeead9b4e49262"
format_stamp: "Formatted at 2026-09-23 21:05:47 on dist-test-slave-8xn5"
server_key: "3957ee10afcd6e135943bb360fe5b1fa"
server_key_iv: "17084d59306d1b53f276dbeff3b7d435"
server_key_version: "encryptionkey@0"
I20260923 21:05:47.928143 20935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:05:47.932343 20950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:47.933847 20935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 21:05:47.934070 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "0f19c083c853480983aeead9b4e49262"
format_stamp: "Formatted at 2026-09-23 21:05:47 on dist-test-slave-8xn5"
server_key: "3957ee10afcd6e135943bb360fe5b1fa"
server_key_iv: "17084d59306d1b53f276dbeff3b7d435"
server_key_version: "encryptionkey@0"
I20260923 21:05:47.934305 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:47.956709 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:47.961793 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:47.962129 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:47.982654 20935 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42555
I20260923 21:05:47.982687 21001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42555 every 8 connection(s)
I20260923 21:05:47.984122 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:47.988313 21002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:47.990592 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 20935
I20260923 21:05:47.990783 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:47.991396 388 external_mini_cluster.cc:1428] Setting key 137dc43a85e744397369911c25cf9bd0
I20260923 21:05:48.003592 21002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262: Bootstrap starting.
I20260923 21:05:48.005497 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:47.993339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55023 (local address 127.0.0.1:42555)
0923 21:05:47.994179 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:47.994192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:47.994234 (+ 42us) server_negotiation.cc:407] Connection header received
0923 21:05:47.996026 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:47.996051 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:47.996499 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:47.996827 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:47.997442 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:47.998808 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:48.000761 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:48.001485 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:48.001626 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:48.003057 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:48.003065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:48.003079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:48.003643 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:48.003798 (+ 155us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:48.003834 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:48.003965 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:48.004255 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 21:05:48.004488 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":399,"threads_started":1}
I20260923 21:05:48.007169 21002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:48.008304 21002 log.cc:824] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:48.009949 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:05:48.011555 21002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262: No bootstrap required, opened a new log
I20260923 21:05:48.022523 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:48.014937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46645 (local address 127.0.0.1:42555)
0923 21:05:48.015162 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:48.015167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:48.015501 (+ 334us) server_negotiation.cc:407] Connection header received
0923 21:05:48.015695 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:48.015699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:48.015764 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:48.015851 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:48.016454 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:48.017234 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:48.019347 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:48.019861 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:48.020001 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:48.021470 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:48.021477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:48.021482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:48.021507 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:48.021571 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:48.021592 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:48.021691 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:48.021937 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 21:05:48.022016 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 21:05:48.022150 21002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f19c083c853480983aeead9b4e49262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260923 21:05:48.022743 21002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:48.022889 21002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f19c083c853480983aeead9b4e49262, State: Initialized, Role: FOLLOWER
I20260923 21:05:48.023573 21002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f19c083c853480983aeead9b4e49262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260923 21:05:48.023850 21002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:48.024024 21002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:48.024232 21002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:48.026584 21002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f19c083c853480983aeead9b4e49262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260923 21:05:48.027153 21002 leader_election.cc:302] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f19c083c853480983aeead9b4e49262; no voters:
I20260923 21:05:48.027781 21002 leader_election.cc:288] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:48.027952 21012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:48.028687 21012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [term 1 LEADER]: Becoming Leader. State: Replica: 0f19c083c853480983aeead9b4e49262, State: Running, Role: LEADER
I20260923 21:05:48.029222 21012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f19c083c853480983aeead9b4e49262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } }
I20260923 21:05:48.029987 21002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:48.032696 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f19c083c853480983aeead9b4e49262. Latest consensus state: current_term: 1 leader_uuid: "0f19c083c853480983aeead9b4e49262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f19c083c853480983aeead9b4e49262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } } }
I20260923 21:05:48.032820 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f19c083c853480983aeead9b4e49262" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f19c083c853480983aeead9b4e49262" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42555 } } }
I20260923 21:05:48.033138 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:48.033203 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:48.034257 21020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:48.041224 21020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:48.051743 21020 catalog_manager.cc:1347] Generated new cluster ID: 95159f7340d14d68a8f25e0c00d9cea9
I20260923 21:05:48.051892 21020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:48.090929 21020 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:48.093515 21020 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:48.120385 21020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f19c083c853480983aeead9b4e49262: Generated new TSK 0
I20260923 21:05:48.121635 21020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:48.135351 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20935
W20260923 21:05:48.136747 20997 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55023
W20260923 21:05:48.137909 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 20935: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:05:48.138128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 20935
W20260923 21:05:48.156250 20920 connection.cc:535] client connection to 127.0.0.1:42555 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42555): Connection reset by peer (error 104)
2026-09-23T21:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2095 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:48 dist-test-slave-8xn5 krb5kdc[21033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:48 dist-test-slave-8xn5 krb5kdc[21033](info): set up 2 sockets
Sep 23 21:05:48 dist-test-slave-8xn5 krb5kdc[21033](info): commencing operation
krb5kdc: starting...
W20260923 21:05:49.330879 388 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:49 dist-test-slave-8xn5 krb5kdc[21033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:49.611069 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:49.957077 21046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:49.957410 21046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:49.957587 21046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:49.965150 21046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:49.965303 21046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:49.965353 21046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:49.965459 21046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:49.965555 21046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:49.976432 21046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:49.977964 21046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:49.979509 21046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:49.992722 21051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:49.992622 21054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:49.992654 21052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:49.993718 21046 server_base.cc:1034] running on GCE node
I20260923 21:05:49.994743 21046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:49.996415 21046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:49.997669 21046 hybrid_clock.cc:648] HybridClock initialized: now 1790197549997585 us; error 73 us; skew 500 ppm
I20260923 21:05:49.998100 21046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:49 dist-test-slave-8xn5 krb5kdc[21033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:50.001863 21046 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:05:50.003626 21046 webserver.cc:460] Webserver started at http://127.0.0.1:44243/ using document root <none> and password file <none>
I20260923 21:05:50.004413 21046 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:50.004531 21046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:50.004864 21046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:50.008419 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "32a79757d0d0406c9d3b46c0944b75f3"
format_stamp: "Formatted at 2026-09-23 21:05:49 on dist-test-slave-8xn5"
server_key: "25430ed77a1d173c9b85806f7de31028"
server_key_iv: "cb3419c100730587a7574c2e49ff1425"
server_key_version: "encryptionkey@0"
I20260923 21:05:50.009290 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "32a79757d0d0406c9d3b46c0944b75f3"
format_stamp: "Formatted at 2026-09-23 21:05:49 on dist-test-slave-8xn5"
server_key: "25430ed77a1d173c9b85806f7de31028"
server_key_iv: "cb3419c100730587a7574c2e49ff1425"
server_key_version: "encryptionkey@0"
I20260923 21:05:50.015765 21046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 21:05:50.020371 21061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:50.022189 21046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 21:05:50.022483 21046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "32a79757d0d0406c9d3b46c0944b75f3"
format_stamp: "Formatted at 2026-09-23 21:05:49 on dist-test-slave-8xn5"
server_key: "25430ed77a1d173c9b85806f7de31028"
server_key_iv: "cb3419c100730587a7574c2e49ff1425"
server_key_version: "encryptionkey@0"
I20260923 21:05:50.022766 21046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:50.029435 21046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:50.034747 21046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:50.035104 21046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:50.055027 21046 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42613
I20260923 21:05:50.055083 21112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42613 every 8 connection(s)
I20260923 21:05:50.056581 21046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:50.058197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21046
I20260923 21:05:50.058439 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:50.059043 388 external_mini_cluster.cc:1428] Setting key 0f6924fd50373d16b1afaa4557c93a02
I20260923 21:05:50.062016 21113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:50.067694 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:50.061146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45551 (local address 127.0.0.1:42613)
0923 21:05:50.061981 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:50.061993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:50.062039 (+ 46us) server_negotiation.cc:407] Connection header received
0923 21:05:50.063975 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:50.064000 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:50.064365 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:50.064838 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:50.065954 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:50.065959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:50.065974 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:50.065990 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:50.066075 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:50.066096 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:50.066368 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:50.066521 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 21:05:50.066711 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":379,"threads_started":1}
I20260923 21:05:50.071969 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:05:50.077036 21113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3: Bootstrap starting.
I20260923 21:05:50.081156 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:50.078446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41079 (local address 127.0.0.1:42613)
0923 21:05:50.078675 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:50.078681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:50.079013 (+ 332us) server_negotiation.cc:407] Connection header received
0923 21:05:50.079229 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:50.079233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:50.079291 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:50.079407 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:50.080454 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:50.080457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:50.080461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:50.080476 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:50.080515 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:50.080530 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:50.080623 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:50.080779 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 21:05:50.080831 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 21:05:50.081513 21113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:50.083041 21113 log.cc:824] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:50.086896 21113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3: No bootstrap required, opened a new log
I20260923 21:05:50.094864 21113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a79757d0d0406c9d3b46c0944b75f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260923 21:05:50.095404 21113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:50.095531 21113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a79757d0d0406c9d3b46c0944b75f3, State: Initialized, Role: FOLLOWER
I20260923 21:05:50.096235 21113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a79757d0d0406c9d3b46c0944b75f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260923 21:05:50.096496 21113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:50.096660 21113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:50.096853 21113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:50.099588 21113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a79757d0d0406c9d3b46c0944b75f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260923 21:05:50.100186 21113 leader_election.cc:302] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a79757d0d0406c9d3b46c0944b75f3; no voters:
I20260923 21:05:50.100858 21113 leader_election.cc:288] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:50.101054 21123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:50.101830 21123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [term 1 LEADER]: Becoming Leader. State: Replica: 32a79757d0d0406c9d3b46c0944b75f3, State: Running, Role: LEADER
I20260923 21:05:50.102563 21123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a79757d0d0406c9d3b46c0944b75f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } }
I20260923 21:05:50.103219 21113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:50.106213 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32a79757d0d0406c9d3b46c0944b75f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a79757d0d0406c9d3b46c0944b75f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } } }
I20260923 21:05:50.106156 21125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a79757d0d0406c9d3b46c0944b75f3. Latest consensus state: current_term: 1 leader_uuid: "32a79757d0d0406c9d3b46c0944b75f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a79757d0d0406c9d3b46c0944b75f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42613 } } }
I20260923 21:05:50.106717 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:50.106757 21125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:50.113974 21130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:50.120381 21130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:50.129484 21130 catalog_manager.cc:1347] Generated new cluster ID: 56f28b0293744b3b83611e8b19799f46
I20260923 21:05:50.129628 21130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:50.149327 21130 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:50.149821 21130 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:50.163801 21130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32a79757d0d0406c9d3b46c0944b75f3: Generated new TSK 0
I20260923 21:05:50.165055 21130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:50.177465 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21046
W20260923 21:05:50.178766 21108 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45551
W20260923 21:05:50.179522 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21046: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:05:50.179713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21046
2026-09-23T21:05:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:50 dist-test-slave-8xn5 krb5kdc[21144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:50 dist-test-slave-8xn5 krb5kdc[21144](info): set up 2 sockets
Sep 23 21:05:50 dist-test-slave-8xn5 krb5kdc[21144](info): commencing operation
krb5kdc: starting...
W20260923 21:05:51.294840 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:51 dist-test-slave-8xn5 krb5kdc[21144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:51.568271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:51.924703 21157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:51.925031 21157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:51.925133 21157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:51.932961 21157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:51.933105 21157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:51.933161 21157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:51.933223 21157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:51.933275 21157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:51.944175 21157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:51.945628 21157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:51.947081 21157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:51.960507 21163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:51.960618 21165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:51.960539 21162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:51.961517 21157 server_base.cc:1034] running on GCE node
I20260923 21:05:51.962554 21157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:51.964221 21157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:51.965463 21157 hybrid_clock.cc:648] HybridClock initialized: now 1790197551965421 us; error 68 us; skew 500 ppm
I20260923 21:05:51.965834 21157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:51 dist-test-slave-8xn5 krb5kdc[21144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:51.969172 21157 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:05:51.970813 21157 webserver.cc:460] Webserver started at http://127.0.0.1:36053/ using document root <none> and password file <none>
I20260923 21:05:51.971508 21157 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:51.971611 21157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:51.971866 21157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:51.975190 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "cb99b10e47774295874111e5de6c986f"
format_stamp: "Formatted at 2026-09-23 21:05:51 on dist-test-slave-8xn5"
server_key: "16c63dbc562d6d59e6f952e4c8271fe0"
server_key_iv: "a0a4e9982fb792647e46ffe0cdc47e6f"
server_key_version: "encryptionkey@0"
I20260923 21:05:51.975994 21157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "cb99b10e47774295874111e5de6c986f"
format_stamp: "Formatted at 2026-09-23 21:05:51 on dist-test-slave-8xn5"
server_key: "16c63dbc562d6d59e6f952e4c8271fe0"
server_key_iv: "a0a4e9982fb792647e46ffe0cdc47e6f"
server_key_version: "encryptionkey@0"
I20260923 21:05:51.981974 21157 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 21:05:51.986255 21172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:51.988013 21157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260923 21:05:51.988238 21157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "cb99b10e47774295874111e5de6c986f"
format_stamp: "Formatted at 2026-09-23 21:05:51 on dist-test-slave-8xn5"
server_key: "16c63dbc562d6d59e6f952e4c8271fe0"
server_key_iv: "a0a4e9982fb792647e46ffe0cdc47e6f"
server_key_version: "encryptionkey@0"
I20260923 21:05:51.988476 21157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:52.002563 21157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:52.007227 21157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:52.007550 21157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:52.027139 21157 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43231
I20260923 21:05:52.027168 21223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43231 every 8 connection(s)
I20260923 21:05:52.028630 21157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:52.032790 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:52.034780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21157
I20260923 21:05:52.034968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:52.035427 388 external_mini_cluster.cc:1428] Setting key 3cec17967c074773ccd378cee20d35ca
I20260923 21:05:52.048045 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f: Bootstrap starting.
Sep 23 21:05:52 dist-test-slave-8xn5 krb5kdc[21144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 21:05:52.052098 21224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:52.053578 21224 log.cc:824] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:52.057386 21224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f: No bootstrap required, opened a new log
I20260923 21:05:52.061510 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:52.037058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55841 (local address 127.0.0.1:43231)
0923 21:05:52.037902 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:52.037914 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:52.037961 (+ 47us) server_negotiation.cc:407] Connection header received
0923 21:05:52.039757 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:52.039784 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:52.040255 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:52.040632 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:52.042472 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:52.043821 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:52.045844 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:52.047042 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:52.047207 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:52.052207 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:52.052219 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:52.052240 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:52.052314 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:52.056611 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:52.057404 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:52.057415 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:52.057423 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:52.057630 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:52.058095 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:52.058102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:52.058107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:52.058864 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:52.059163 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:52.059575 (+ 412us) server_negotiation.cc:299] Negotiation successful
0923 21:05:52.060085 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":385,"threads_started":1}
I20260923 21:05:52.065641 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:05:52.067359 21224 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb99b10e47774295874111e5de6c986f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260923 21:05:52.067750 21224 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:52.067858 21224 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb99b10e47774295874111e5de6c986f, State: Initialized, Role: FOLLOWER
I20260923 21:05:52.068527 21224 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb99b10e47774295874111e5de6c986f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260923 21:05:52.068820 21224 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:52.068966 21224 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:52.069149 21224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:52.072140 21224 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb99b10e47774295874111e5de6c986f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260923 21:05:52.072722 21224 leader_election.cc:302] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb99b10e47774295874111e5de6c986f; no voters:
I20260923 21:05:52.073448 21224 leader_election.cc:288] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:52.073570 21234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:52.074826 21234 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [term 1 LEADER]: Becoming Leader. State: Replica: cb99b10e47774295874111e5de6c986f, State: Running, Role: LEADER
I20260923 21:05:52.075554 21234 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb99b10e47774295874111e5de6c986f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } }
I20260923 21:05:52.076148 21224 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:52.078797 21235 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb99b10e47774295874111e5de6c986f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb99b10e47774295874111e5de6c986f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } } }
I20260923 21:05:52.079268 21235 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:52.080987 21236 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb99b10e47774295874111e5de6c986f. Latest consensus state: current_term: 1 leader_uuid: "cb99b10e47774295874111e5de6c986f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb99b10e47774295874111e5de6c986f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43231 } } }
I20260923 21:05:52.081420 21236 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:52.088812 21239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:52.089327 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:52.070288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45201 (local address 127.0.0.1:43231)
0923 21:05:52.070551 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:52.070559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:52.070928 (+ 369us) server_negotiation.cc:407] Connection header received
0923 21:05:52.071135 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:52.071142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:52.071218 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:52.071336 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:52.072813 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:52.073716 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:52.075568 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:52.076139 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:52.076288 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:52.079639 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:52.079648 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:52.079655 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:52.079698 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:52.084591 (+ 4893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:52.085944 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:52.085953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:52.085960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:52.086169 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:52.086818 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:52.086827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:52.086837 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:52.087441 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:52.087768 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:52.088788 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0923 21:05:52.089029 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260923 21:05:52.096011 21239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:52.106175 21239 catalog_manager.cc:1347] Generated new cluster ID: 9ba04f9ef23c4673a51bc1c8b4afdf4d
I20260923 21:05:52.106415 21239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:52.128872 21239 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:52.130282 21239 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:52.140746 21239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb99b10e47774295874111e5de6c986f: Generated new TSK 0
I20260923 21:05:52.141773 21239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:52.149672 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21157
I20260923 21:05:52.151064 21219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:52.250674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21157
2026-09-23T21:05:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2071 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:52 dist-test-slave-8xn5 krb5kdc[21256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:52 dist-test-slave-8xn5 krb5kdc[21256](info): set up 2 sockets
Sep 23 21:05:52 dist-test-slave-8xn5 krb5kdc[21256](info): commencing operation
krb5kdc: starting...
W20260923 21:05:53.394845 388 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:53 dist-test-slave-8xn5 krb5kdc[21256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:53.679152 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:54.026103 21269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:54.026485 21269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:54.026662 21269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:54.034317 21269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:54.034498 21269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:54.034590 21269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:54.034716 21269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:54.034783 21269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:54.045234 21269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:54.046746 21269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:54.048142 21269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:54.060850 21274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:54.060776 21275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:54.060925 21277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:54.061753 21269 server_base.cc:1034] running on GCE node
I20260923 21:05:54.062671 21269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:54.064250 21269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:54.065467 21269 hybrid_clock.cc:648] HybridClock initialized: now 1790197554065408 us; error 57 us; skew 500 ppm
I20260923 21:05:54.065855 21269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:54 dist-test-slave-8xn5 krb5kdc[21256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:54.072211 21269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:05:54.073784 21269 webserver.cc:460] Webserver started at http://127.0.0.1:42813/ using document root <none> and password file <none>
I20260923 21:05:54.074608 21269 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:54.074752 21269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:54.075039 21269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:54.078147 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1"
format_stamp: "Formatted at 2026-09-23 21:05:54 on dist-test-slave-8xn5"
server_key: "5a246c4a841cd168aae1975dd635781d"
server_key_iv: "c9550be870925e1933dc07783262161c"
server_key_version: "encryptionkey@0"
I20260923 21:05:54.079012 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1"
format_stamp: "Formatted at 2026-09-23 21:05:54 on dist-test-slave-8xn5"
server_key: "5a246c4a841cd168aae1975dd635781d"
server_key_iv: "c9550be870925e1933dc07783262161c"
server_key_version: "encryptionkey@0"
I20260923 21:05:54.084578 21269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:05:54.088737 21284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:54.090533 21269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:05:54.090770 21269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1"
format_stamp: "Formatted at 2026-09-23 21:05:54 on dist-test-slave-8xn5"
server_key: "5a246c4a841cd168aae1975dd635781d"
server_key_iv: "c9550be870925e1933dc07783262161c"
server_key_version: "encryptionkey@0"
I20260923 21:05:54.091013 21269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:54.124594 21269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:54.129487 21269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:54.129938 21269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:54.151018 21269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36537
I20260923 21:05:54.151042 21335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36537 every 8 connection(s)
I20260923 21:05:54.152539 21269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:54.156621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21269
I20260923 21:05:54.156708 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:54.156831 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:54.157370 388 external_mini_cluster.cc:1428] Setting key 700e4660ae36fb4280cbbd77fc1f5237
I20260923 21:05:54.170164 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1: Bootstrap starting.
I20260923 21:05:54.171255 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:54.159393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37323 (local address 127.0.0.1:36537)
0923 21:05:54.160320 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:54.160332 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:54.160378 (+ 46us) server_negotiation.cc:407] Connection header received
0923 21:05:54.162005 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:54.162030 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:54.162558 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:54.163012 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:54.163604 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:54.164930 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:54.166366 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:54.167096 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:54.167329 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:54.168782 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:54.168792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:54.168808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:54.169405 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:54.169484 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:54.169510 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:54.169651 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:54.169889 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 21:05:54.170175 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":416,"threads_started":1}
I20260923 21:05:54.173694 21336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:54.175143 21336 log.cc:824] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:54.175595 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:05:54.179154 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1: No bootstrap required, opened a new log
I20260923 21:05:54.188103 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:54.181015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40549 (local address 127.0.0.1:36537)
0923 21:05:54.181308 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:54.181316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:54.181707 (+ 391us) server_negotiation.cc:407] Connection header received
0923 21:05:54.181924 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:54.181928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:54.181998 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:54.182079 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:54.182786 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:54.183547 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:54.185523 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:54.185997 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:54.186097 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:54.187309 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:54.187314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:54.187318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:54.187336 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:54.187375 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:54.187391 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:54.187464 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:54.187618 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 21:05:54.187709 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:05:54.190408 21336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36537 } }
I20260923 21:05:54.190869 21336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:54.190987 21336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740ce9e63a6e4a4daa70ef71ff9a0bc1, State: Initialized, Role: FOLLOWER
I20260923 21:05:54.191594 21336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36537 } }
I20260923 21:05:54.191857 21336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:54.192018 21336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:54.192287 21336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:54.195262 21336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36537 } }
I20260923 21:05:54.195874 21336 leader_election.cc:302] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 740ce9e63a6e4a4daa70ef71ff9a0bc1; no voters:
I20260923 21:05:54.196524 21336 leader_election.cc:288] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:54.196734 21346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:54.197537 21346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [term 1 LEADER]: Becoming Leader. State: Replica: 740ce9e63a6e4a4daa70ef71ff9a0bc1, State: Running, Role: LEADER
I20260923 21:05:54.198144 21346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36537 } }
I20260923 21:05:54.198818 21336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:54.202965 21348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 740ce9e63a6e4a4daa70ef71ff9a0bc1. Latest consensus state: current_term: 1 leader_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36537 } } }
I20260923 21:05:54.203413 21348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:54.204030 21347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740ce9e63a6e4a4daa70ef71ff9a0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36537 } } }
I20260923 21:05:54.204506 21347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:54.209666 21354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:54.216233 21354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:54.227490 21354 catalog_manager.cc:1347] Generated new cluster ID: 5e9d20ea715f4d42b99e1a8cbe991204
I20260923 21:05:54.227638 21354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:54.240471 21354 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:54.241901 21354 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:54.254993 21354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 740ce9e63a6e4a4daa70ef71ff9a0bc1: Generated new TSK 0
I20260923 21:05:54.255930 21354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:54.266052 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21269
W20260923 21:05:54.267549 21331 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37323
W20260923 21:05:54.268343 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21269: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:05:54.268498 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21269
2026-09-23T21:05:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2017 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:54 dist-test-slave-8xn5 krb5kdc[21367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:54 dist-test-slave-8xn5 krb5kdc[21367](info): set up 2 sockets
Sep 23 21:05:54 dist-test-slave-8xn5 krb5kdc[21367](info): commencing operation
krb5kdc: starting...
W20260923 21:05:55.422873 388 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:55 dist-test-slave-8xn5 krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:55.701925 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:56.049657 21380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:56.050011 21380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:56.050187 21380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:56.057832 21380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:56.057965 21380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:56.058017 21380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:56.058105 21380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:56.058177 21380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:56.068650 21380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:56.070040 21380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:56.071498 21380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:56.084260 21388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:56.084338 21385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:56.084365 21386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:56.085602 21380 server_base.cc:1034] running on GCE node
I20260923 21:05:56.086541 21380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:56.088162 21380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:56.089402 21380 hybrid_clock.cc:648] HybridClock initialized: now 1790197556089356 us; error 46 us; skew 500 ppm
I20260923 21:05:56.089776 21380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:56 dist-test-slave-8xn5 krb5kdc[21367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:56.093379 21380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:05:56.095723 21380 webserver.cc:460] Webserver started at http://127.0.0.1:44413/ using document root <none> and password file <none>
I20260923 21:05:56.096581 21380 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:56.096762 21380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:56.097168 21380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:56.100821 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "e7cbd322dcc940f3a38821ddacb00704"
format_stamp: "Formatted at 2026-09-23 21:05:56 on dist-test-slave-8xn5"
server_key: "df74bc17e57c9254803ed622b2f4f7d7"
server_key_iv: "1949f03f736a9201bc8b78bf005da033"
server_key_version: "encryptionkey@0"
I20260923 21:05:56.101831 21380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "e7cbd322dcc940f3a38821ddacb00704"
format_stamp: "Formatted at 2026-09-23 21:05:56 on dist-test-slave-8xn5"
server_key: "df74bc17e57c9254803ed622b2f4f7d7"
server_key_iv: "1949f03f736a9201bc8b78bf005da033"
server_key_version: "encryptionkey@0"
I20260923 21:05:56.108343 21380 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 21:05:56.112600 21395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:56.114143 21380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 21:05:56.114406 21380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "e7cbd322dcc940f3a38821ddacb00704"
format_stamp: "Formatted at 2026-09-23 21:05:56 on dist-test-slave-8xn5"
server_key: "df74bc17e57c9254803ed622b2f4f7d7"
server_key_iv: "1949f03f736a9201bc8b78bf005da033"
server_key_version: "encryptionkey@0"
I20260923 21:05:56.114686 21380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:56.120718 21380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:56.125162 21380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:56.125483 21380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:56.145087 21380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35297
I20260923 21:05:56.145183 21446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35297 every 8 connection(s)
I20260923 21:05:56.146612 21380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:56.149935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21380
I20260923 21:05:56.150188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:56.150707 388 external_mini_cluster.cc:1428] Setting key f55e963dcf56b87eaa14fc0898deddfd
I20260923 21:05:56.150713 21447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:05:56.159931 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:56.152685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59645 (local address 127.0.0.1:35297)
0923 21:05:56.153566 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:56.153579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:56.153627 (+ 48us) server_negotiation.cc:407] Connection header received
0923 21:05:56.155647 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:56.155673 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:56.156103 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:56.156852 (+ 749us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:56.158062 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:56.158069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:56.158083 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:56.158098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:56.158170 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:56.158191 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:56.158449 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:56.158693 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 21:05:56.158935 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":396,"threads_started":1}
I20260923 21:05:56.164095 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:05:56.165591 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704: Bootstrap starting.
I20260923 21:05:56.169435 21447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:56.170697 21447 log.cc:824] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:56.171157 21450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:56.168592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58077 (local address 127.0.0.1:35297)
0923 21:05:56.168775 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:56.168781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:56.169091 (+ 310us) server_negotiation.cc:407] Connection header received
0923 21:05:56.169240 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:56.169248 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:56.169325 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:56.169459 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:56.170457 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:56.170468 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:56.170475 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:05:56.170498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:56.170545 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:05:56.170565 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:56.170695 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:56.170821 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 21:05:56.170882 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:05:56.174304 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704: No bootstrap required, opened a new log
I20260923 21:05:56.182672 21447 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbd322dcc940f3a38821ddacb00704" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260923 21:05:56.183120 21447 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:56.183261 21447 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7cbd322dcc940f3a38821ddacb00704, State: Initialized, Role: FOLLOWER
I20260923 21:05:56.183913 21447 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbd322dcc940f3a38821ddacb00704" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260923 21:05:56.184182 21447 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:56.184342 21447 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:56.184518 21447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:56.187384 21447 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbd322dcc940f3a38821ddacb00704" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260923 21:05:56.187956 21447 leader_election.cc:302] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7cbd322dcc940f3a38821ddacb00704; no voters:
I20260923 21:05:56.188561 21447 leader_election.cc:288] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:56.188762 21457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:56.189837 21457 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [term 1 LEADER]: Becoming Leader. State: Replica: e7cbd322dcc940f3a38821ddacb00704, State: Running, Role: LEADER
I20260923 21:05:56.190486 21457 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbd322dcc940f3a38821ddacb00704" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260923 21:05:56.191113 21447 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:56.193495 21459 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7cbd322dcc940f3a38821ddacb00704. Latest consensus state: current_term: 1 leader_uuid: "e7cbd322dcc940f3a38821ddacb00704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbd322dcc940f3a38821ddacb00704" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260923 21:05:56.193923 21459 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:56.193648 21458 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7cbd322dcc940f3a38821ddacb00704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7cbd322dcc940f3a38821ddacb00704" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260923 21:05:56.194060 21458 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:56.200997 21463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:56.207304 21463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:56.226080 21463 catalog_manager.cc:1347] Generated new cluster ID: 8e95d0a43f1e46b486ceef2aa4d8d984
I20260923 21:05:56.226218 21463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:56.267010 21463 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:56.267506 21463 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:56.275979 21463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7cbd322dcc940f3a38821ddacb00704: Generated new TSK 0
I20260923 21:05:56.276932 21463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:56.291374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21380
W20260923 21:05:56.292761 21442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59645
W20260923 21:05:56.293663 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:05:56.293829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21380
2026-09-23T21:05:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2025 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:56 dist-test-slave-8xn5 krb5kdc[21478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:56 dist-test-slave-8xn5 krb5kdc[21478](info): set up 2 sockets
Sep 23 21:05:56 dist-test-slave-8xn5 krb5kdc[21478](info): commencing operation
krb5kdc: starting...
W20260923 21:05:57.418962 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:57 dist-test-slave-8xn5 krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:57.699033 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:05:58.047971 21491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:05:58.048362 21491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:05:58.048532 21491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:05:58.056218 21491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:05:58.056396 21491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:05:58.056514 21491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:05:58.056624 21491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:05:58.056705 21491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:05:58.067023 21491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:05:58.068447 21491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:05:58.069804 21491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:05:58.083222 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:58.083303 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:05:58.083525 21499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:05:58.084652 21491 server_base.cc:1034] running on GCE node
I20260923 21:05:58.085484 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:05:58.087136 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:05:58.088363 21491 hybrid_clock.cc:648] HybridClock initialized: now 1790197558088311 us; error 47 us; skew 500 ppm
I20260923 21:05:58.088943 21491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:05:58 dist-test-slave-8xn5 krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:05:58.092427 21491 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:05:58.094126 21491 webserver.cc:460] Webserver started at http://127.0.0.1:38561/ using document root <none> and password file <none>
I20260923 21:05:58.094959 21491 fs_manager.cc:359] Metadata directory not provided
I20260923 21:05:58.095103 21491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:05:58.095454 21491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:05:58.098829 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92"
format_stamp: "Formatted at 2026-09-23 21:05:58 on dist-test-slave-8xn5"
server_key: "89707b1fce87aca41ad3385777c041ca"
server_key_iv: "b78750f2dc3989b793c682098c4b971e"
server_key_version: "encryptionkey@0"
I20260923 21:05:58.099723 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92"
format_stamp: "Formatted at 2026-09-23 21:05:58 on dist-test-slave-8xn5"
server_key: "89707b1fce87aca41ad3385777c041ca"
server_key_iv: "b78750f2dc3989b793c682098c4b971e"
server_key_version: "encryptionkey@0"
I20260923 21:05:58.105635 21491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 21:05:58.110098 21506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:05:58.111709 21491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:05:58.111934 21491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92"
format_stamp: "Formatted at 2026-09-23 21:05:58 on dist-test-slave-8xn5"
server_key: "89707b1fce87aca41ad3385777c041ca"
server_key_iv: "b78750f2dc3989b793c682098c4b971e"
server_key_version: "encryptionkey@0"
I20260923 21:05:58.112175 21491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:05:58.128139 21491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:05:58.132639 21491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:05:58.132949 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:05:58.152602 21491 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41253
I20260923 21:05:58.152647 21557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41253 every 8 connection(s)
I20260923 21:05:58.154121 21491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:05:58.156740 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21491
I20260923 21:05:58.156921 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:05:58.157452 388 external_mini_cluster.cc:1428] Setting key a35a5135e4ad868e30f9127d5dea6be0
I20260923 21:05:58.158834 21558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 21:05:58 dist-test-slave-8xn5 krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 21:05:58.174276 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92: Bootstrap starting.
I20260923 21:05:58.178004 21558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92: Neither blocks nor log segments found. Creating new log.
I20260923 21:05:58.179224 21558 log.cc:824] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92: Log is configured to *not* fsync() on all Append() calls
I20260923 21:05:58.182440 21558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92: No bootstrap required, opened a new log
I20260923 21:05:58.182978 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:58.159610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:56923 (local address 127.0.0.1:41253)
0923 21:05:58.160518 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:58.160531 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:58.160578 (+ 47us) server_negotiation.cc:407] Connection header received
0923 21:05:58.162592 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:58.162618 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:58.163111 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:58.163521 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:58.165363 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:58.166905 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:58.168695 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:58.169429 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:58.169591 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:58.174894 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:58.174903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:58.174920 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:58.174968 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:58.178450 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:58.179353 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:58.179364 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:58.179397 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:58.179597 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:58.180081 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:58.180087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:58.180090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:58.180666 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:58.181013 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:58.181504 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 21:05:58.181837 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":411,"threads_started":1}
I20260923 21:05:58.187132 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:05:58.190850 21558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41253 } }
I20260923 21:05:58.191355 21558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:05:58.191471 21558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1dc3ebcd3c4e37a3c5d584c64b5c92, State: Initialized, Role: FOLLOWER
I20260923 21:05:58.192279 21558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41253 } }
I20260923 21:05:58.192600 21558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:05:58.192759 21558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:05:58.192981 21558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:05:58.196317 21558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41253 } }
I20260923 21:05:58.196997 21558 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d1dc3ebcd3c4e37a3c5d584c64b5c92; no voters:
I20260923 21:05:58.197703 21558 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:05:58.197814 21568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:05:58.200702 21558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:05:58.204989 21568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1dc3ebcd3c4e37a3c5d584c64b5c92, State: Running, Role: LEADER
I20260923 21:05:58.205807 21568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41253 } }
I20260923 21:05:58.209082 21569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41253 } } }
I20260923 21:05:58.209650 21569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:58.213790 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:05:58.192172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:54059 (local address 127.0.0.1:41253)
0923 21:05:58.192438 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 21:05:58.192446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:05:58.192833 (+ 387us) server_negotiation.cc:407] Connection header received
0923 21:05:58.193031 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:05:58.193035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:05:58.193127 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:05:58.193225 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:05:58.195150 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:58.195990 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:58.197926 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:05:58.198991 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:05:58.199158 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:05:58.203316 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:05:58.203327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:05:58.203335 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:05:58.203394 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:05:58.208088 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:58.209428 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:58.209434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:58.209438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:58.209719 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:05:58.210516 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:05:58.210524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:05:58.210531 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:05:58.211060 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:05:58.212708 (+ 1648us) server_negotiation.cc:1035] Waiting for connection context
0923 21:05:58.212957 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 21:05:58.213222 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 21:05:58.221477 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1dc3ebcd3c4e37a3c5d584c64b5c92. Latest consensus state: current_term: 1 leader_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1dc3ebcd3c4e37a3c5d584c64b5c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41253 } } }
I20260923 21:05:58.221988 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92 [sys.catalog]: This master's current role is: LEADER
I20260923 21:05:58.223242 21578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:05:58.229377 21578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:05:58.239420 21578 catalog_manager.cc:1347] Generated new cluster ID: 2110f65763144ecfa9fcbc57e39a176a
I20260923 21:05:58.239560 21578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:05:58.258541 21578 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:05:58.260182 21578 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:05:58.266387 21578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d1dc3ebcd3c4e37a3c5d584c64b5c92: Generated new TSK 0
I20260923 21:05:58.267530 21578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:05:58.285233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21491
I20260923 21:05:58.286613 21553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:05:58.380441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21491
W20260923 21:05:58.399921 21476 connection.cc:535] client connection to 127.0.0.1:41253 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41253): Connection reset by peer (error 104)
2026-09-23T21:05:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:05:58 dist-test-slave-8xn5 krb5kdc[21591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:05:58 dist-test-slave-8xn5 krb5kdc[21591](info): set up 2 sockets
Sep 23 21:05:58 dist-test-slave-8xn5 krb5kdc[21591](info): commencing operation
krb5kdc: starting...
W20260923 21:05:59.507228 388 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:05:59 dist-test-slave-8xn5 krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:05:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:05:59.783380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:00.133603 21604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:00.133941 21604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:00.134040 21604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:00.141640 21604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:00.141776 21604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:00.141825 21604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:00.141880 21604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:00.141927 21604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:00.152324 21604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:00.153676 21604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:00.155104 21604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:00.168318 21609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:00.168267 21610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:00.168236 21612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:00.169740 21604 server_base.cc:1034] running on GCE node
I20260923 21:06:00.170755 21604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:00.172343 21604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:00.173597 21604 hybrid_clock.cc:648] HybridClock initialized: now 1790197560173552 us; error 44 us; skew 500 ppm
I20260923 21:06:00.173982 21604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:00 dist-test-slave-8xn5 krb5kdc[21591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:00.183275 21604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:06:00.185345 21604 webserver.cc:460] Webserver started at http://127.0.0.1:39657/ using document root <none> and password file <none>
I20260923 21:06:00.186178 21604 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:00.186396 21604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:00.186745 21604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:00.190132 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "1a7a403272144817a4b2a29055826942"
format_stamp: "Formatted at 2026-09-23 21:06:00 on dist-test-slave-8xn5"
server_key: "9c50f631552d915d08b45f787982abf3"
server_key_iv: "8adaaeb03deb87ffcd76945f53a6d242"
server_key_version: "encryptionkey@0"
I20260923 21:06:00.191123 21604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "1a7a403272144817a4b2a29055826942"
format_stamp: "Formatted at 2026-09-23 21:06:00 on dist-test-slave-8xn5"
server_key: "9c50f631552d915d08b45f787982abf3"
server_key_iv: "8adaaeb03deb87ffcd76945f53a6d242"
server_key_version: "encryptionkey@0"
I20260923 21:06:00.197117 21604 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260923 21:06:00.201345 21619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:00.203140 21604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260923 21:06:00.203397 21604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "1a7a403272144817a4b2a29055826942"
format_stamp: "Formatted at 2026-09-23 21:06:00 on dist-test-slave-8xn5"
server_key: "9c50f631552d915d08b45f787982abf3"
server_key_iv: "8adaaeb03deb87ffcd76945f53a6d242"
server_key_version: "encryptionkey@0"
I20260923 21:06:00.203644 21604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:00.229171 21604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:00.234014 21604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:00.234375 21604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:00.253959 21604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41693
I20260923 21:06:00.253984 21670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41693 every 8 connection(s)
I20260923 21:06:00.255479 21604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:00.259495 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:00.261855 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21604
I20260923 21:06:00.262058 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:00.262639 388 external_mini_cluster.cc:1428] Setting key b67adc1b7f07bb77229e755253a881d9
I20260923 21:06:00.272886 21671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942: Bootstrap starting.
I20260923 21:06:00.276831 21671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:00.277529 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:00.264618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:57691 (local address 127.0.0.1:41693)
0923 21:06:00.265499 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:00.265512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:00.265560 (+ 48us) server_negotiation.cc:407] Connection header received
0923 21:06:00.267359 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:00.267384 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:00.267829 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:00.268183 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:00.268840 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:00.270306 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:00.272317 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:00.273179 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:00.273346 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:00.274808 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:00.274817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:00.274837 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:00.275544 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:00.275636 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:00.275665 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:00.275814 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:00.276195 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 21:06:00.276395 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":413,"threads_started":1}
I20260923 21:06:00.278012 21671 log.cc:824] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:00.281464 21671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942: No bootstrap required, opened a new log
I20260923 21:06:00.282238 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:00.291620 21671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7a403272144817a4b2a29055826942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260923 21:06:00.292107 21671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:00.292256 21671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a7a403272144817a4b2a29055826942, State: Initialized, Role: FOLLOWER
I20260923 21:06:00.292986 21671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7a403272144817a4b2a29055826942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260923 21:06:00.293251 21671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:00.293408 21671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:00.293607 21671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:00.294955 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:00.288081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:38397 (local address 127.0.0.1:41693)
0923 21:06:00.288366 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:00.288377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:00.288850 (+ 473us) server_negotiation.cc:407] Connection header received
0923 21:06:00.289063 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:00.289068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:00.289134 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:00.289254 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:00.289809 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:00.290671 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:00.292050 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:00.292652 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:00.292800 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:00.294123 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:00.294129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:00.294133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:00.294163 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:00.294223 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:00.294243 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:00.294410 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:00.294605 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 21:06:00.294670 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 21:06:00.296947 21671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7a403272144817a4b2a29055826942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260923 21:06:00.297498 21671 leader_election.cc:302] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a7a403272144817a4b2a29055826942; no voters:
I20260923 21:06:00.298167 21671 leader_election.cc:288] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:00.298301 21681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:00.299084 21681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [term 1 LEADER]: Becoming Leader. State: Replica: 1a7a403272144817a4b2a29055826942, State: Running, Role: LEADER
I20260923 21:06:00.299738 21681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7a403272144817a4b2a29055826942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260923 21:06:00.300508 21671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:00.303557 21683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a7a403272144817a4b2a29055826942. Latest consensus state: current_term: 1 leader_uuid: "1a7a403272144817a4b2a29055826942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7a403272144817a4b2a29055826942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } } }
I20260923 21:06:00.303987 21683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:00.303692 21682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a7a403272144817a4b2a29055826942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a7a403272144817a4b2a29055826942" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } } }
I20260923 21:06:00.304148 21682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:00.310606 21689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:00.316200 21689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:00.325778 21689 catalog_manager.cc:1347] Generated new cluster ID: 684801f66ba84f828917632fc8f157e3
I20260923 21:06:00.325925 21689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:00.337236 21689 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:00.339246 21689 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:00.350761 21689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a7a403272144817a4b2a29055826942: Generated new TSK 0
I20260923 21:06:00.351964 21689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:00.370783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21604
W20260923 21:06:00.372011 21666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:57691
W20260923 21:06:00.372874 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21604: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:00.373051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21604
W20260923 21:06:00.390375 21589 connection.cc:535] client connection to 127.0.0.1:41693 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41693): Connection reset by peer (error 104)
2026-09-23T21:06:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:00 dist-test-slave-8xn5 krb5kdc[21702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:00 dist-test-slave-8xn5 krb5kdc[21702](info): set up 2 sockets
Sep 23 21:06:00 dist-test-slave-8xn5 krb5kdc[21702](info): commencing operation
krb5kdc: starting...
W20260923 21:06:01.506947 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.010s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:01 dist-test-slave-8xn5 krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:06:01.790202 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:02.160318 21715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:02.160663 21715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:02.160882 21715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:02.168815 21715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:02.168968 21715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:02.169083 21715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:02.169202 21715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:02.169296 21715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:02.180197 21715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:02.181766 21715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:02.183226 21715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:02.197364 21723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:02.197326 21720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:02.197387 21721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:02.199221 21715 server_base.cc:1034] running on GCE node
I20260923 21:06:02.200239 21715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:02.201849 21715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:02.203112 21715 hybrid_clock.cc:648] HybridClock initialized: now 1790197562202941 us; error 156 us; skew 500 ppm
I20260923 21:06:02.203603 21715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:02 dist-test-slave-8xn5 krb5kdc[21702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:02.207787 21715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:06:02.209903 21715 webserver.cc:460] Webserver started at http://127.0.0.1:41871/ using document root <none> and password file <none>
I20260923 21:06:02.211148 21715 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:02.211351 21715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:02.211833 21715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:02.217374 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "00a292102f524ef5bd9bb5fa8a36621f"
format_stamp: "Formatted at 2026-09-23 21:06:02 on dist-test-slave-8xn5"
server_key: "0b1481b64d3fab4dcc628215558efbdb"
server_key_iv: "d47401f5079c36f84387328334d93bc2"
server_key_version: "encryptionkey@0"
I20260923 21:06:02.218865 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "00a292102f524ef5bd9bb5fa8a36621f"
format_stamp: "Formatted at 2026-09-23 21:06:02 on dist-test-slave-8xn5"
server_key: "0b1481b64d3fab4dcc628215558efbdb"
server_key_iv: "d47401f5079c36f84387328334d93bc2"
server_key_version: "encryptionkey@0"
I20260923 21:06:02.226944 21715 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260923 21:06:02.231417 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:02.233407 21715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260923 21:06:02.233662 21715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "00a292102f524ef5bd9bb5fa8a36621f"
format_stamp: "Formatted at 2026-09-23 21:06:02 on dist-test-slave-8xn5"
server_key: "0b1481b64d3fab4dcc628215558efbdb"
server_key_iv: "d47401f5079c36f84387328334d93bc2"
server_key_version: "encryptionkey@0"
I20260923 21:06:02.233939 21715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:02.240928 21715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:02.246466 21715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:02.246788 21715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:02.267292 21715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37653
I20260923 21:06:02.267310 21781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37653 every 8 connection(s)
I20260923 21:06:02.268775 21715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:02.272818 21782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:02.278168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21715
I20260923 21:06:02.278540 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:02.279044 388 external_mini_cluster.cc:1428] Setting key 213eab9c67158167e648a83f7fa4d1f1
I20260923 21:06:02.285655 21782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f: Bootstrap starting.
I20260923 21:06:02.288909 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:02.281154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:34623 (local address 127.0.0.1:37653)
0923 21:06:02.282162 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:02.282176 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:02.282237 (+ 61us) server_negotiation.cc:407] Connection header received
0923 21:06:02.284623 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:02.284653 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:02.285089 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:02.285585 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:02.286947 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:02.286954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:02.286969 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:02.286995 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:02.287060 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:02.287084 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:02.287476 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:02.287667 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 21:06:02.287860 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":415,"threads_started":1}
I20260923 21:06:02.289755 21782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:02.291146 21782 log.cc:824] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:02.293720 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:02.294700 21782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f: No bootstrap required, opened a new log
I20260923 21:06:02.301106 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:02.298431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:58731 (local address 127.0.0.1:37653)
0923 21:06:02.298631 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:02.298640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:02.298989 (+ 349us) server_negotiation.cc:407] Connection header received
0923 21:06:02.299200 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:02.299204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:02.299254 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:02.299373 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:02.300394 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:02.300399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:02.300403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:02.300431 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:02.300493 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:02.300511 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:02.300661 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:02.300745 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 21:06:02.300818 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 21:06:02.304571 21782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a292102f524ef5bd9bb5fa8a36621f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260923 21:06:02.305060 21782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:02.305194 21782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a292102f524ef5bd9bb5fa8a36621f, State: Initialized, Role: FOLLOWER
I20260923 21:06:02.305929 21782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a292102f524ef5bd9bb5fa8a36621f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260923 21:06:02.306232 21782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:02.306422 21782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:02.306629 21782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:02.309311 21782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a292102f524ef5bd9bb5fa8a36621f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260923 21:06:02.309906 21782 leader_election.cc:302] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a292102f524ef5bd9bb5fa8a36621f; no voters:
I20260923 21:06:02.310690 21782 leader_election.cc:288] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:02.310956 21792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:02.311703 21792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [term 1 LEADER]: Becoming Leader. State: Replica: 00a292102f524ef5bd9bb5fa8a36621f, State: Running, Role: LEADER
I20260923 21:06:02.312412 21792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a292102f524ef5bd9bb5fa8a36621f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } }
I20260923 21:06:02.313109 21782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:02.316387 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a292102f524ef5bd9bb5fa8a36621f. Latest consensus state: current_term: 1 leader_uuid: "00a292102f524ef5bd9bb5fa8a36621f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a292102f524ef5bd9bb5fa8a36621f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260923 21:06:02.316365 21793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a292102f524ef5bd9bb5fa8a36621f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a292102f524ef5bd9bb5fa8a36621f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37653 } } }
I20260923 21:06:02.316931 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:02.316944 21793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:02.318513 21799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:02.327016 21799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:02.338583 21799 catalog_manager.cc:1347] Generated new cluster ID: a50106941c4e45e29c4495c37cd96b30
I20260923 21:06:02.338757 21799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:02.360510 21799 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:02.361212 21799 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:02.370635 21799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00a292102f524ef5bd9bb5fa8a36621f: Generated new TSK 0
I20260923 21:06:02.371894 21799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:02.387452 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21715
W20260923 21:06:02.389138 21777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:34623
W20260923 21:06:02.390117 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21715: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:02.390280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21715
2026-09-23T21:06:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:02 dist-test-slave-8xn5 krb5kdc[21813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:02 dist-test-slave-8xn5 krb5kdc[21813](info): set up 2 sockets
Sep 23 21:06:02 dist-test-slave-8xn5 krb5kdc[21813](info): commencing operation
krb5kdc: starting...
W20260923 21:06:03.538869 388 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:03 dist-test-slave-8xn5 krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:06:03.818552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:04.162763 21826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:04.163130 21826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:04.163398 21826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:04.170898 21826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:04.171082 21826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:04.171211 21826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:04.171324 21826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:04.171408 21826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:04.181640 21826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:04.183095 21826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:04.184504 21826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:04.197595 21834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:04.197626 21832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:04.197861 21831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:04.198586 21826 server_base.cc:1034] running on GCE node
I20260923 21:06:04.199434 21826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:04.200951 21826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:04.202208 21826 hybrid_clock.cc:648] HybridClock initialized: now 1790197564202148 us; error 43 us; skew 500 ppm
I20260923 21:06:04.202695 21826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:04 dist-test-slave-8xn5 krb5kdc[21813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:04.206372 21826 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:06:04.208127 21826 webserver.cc:460] Webserver started at http://127.0.0.1:45835/ using document root <none> and password file <none>
I20260923 21:06:04.208858 21826 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:04.208977 21826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:04.209255 21826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:04.214391 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "fd441477e44e45d6aa3eef18f6d66261"
format_stamp: "Formatted at 2026-09-23 21:06:04 on dist-test-slave-8xn5"
server_key: "175798112ff10645882e6cf865e051fc"
server_key_iv: "524bd3f6def2cc2f0897519dd2f6699e"
server_key_version: "encryptionkey@0"
I20260923 21:06:04.215770 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd441477e44e45d6aa3eef18f6d66261"
format_stamp: "Formatted at 2026-09-23 21:06:04 on dist-test-slave-8xn5"
server_key: "175798112ff10645882e6cf865e051fc"
server_key_iv: "524bd3f6def2cc2f0897519dd2f6699e"
server_key_version: "encryptionkey@0"
I20260923 21:06:04.222301 21826 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 21:06:04.226831 21841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:04.228694 21826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260923 21:06:04.229074 21826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "fd441477e44e45d6aa3eef18f6d66261"
format_stamp: "Formatted at 2026-09-23 21:06:04 on dist-test-slave-8xn5"
server_key: "175798112ff10645882e6cf865e051fc"
server_key_iv: "524bd3f6def2cc2f0897519dd2f6699e"
server_key_version: "encryptionkey@0"
I20260923 21:06:04.229352 21826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:04.271651 21826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:04.276690 21826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:04.277042 21826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:04.298239 21826 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43507
I20260923 21:06:04.298291 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43507 every 8 connection(s)
I20260923 21:06:04.299970 21826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:04.303983 21893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:04.305763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21826
I20260923 21:06:04.306035 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:04.306630 388 external_mini_cluster.cc:1428] Setting key 3d7db23b05db2c6fa20446d24fca7bd6
I20260923 21:06:04.317212 21893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261: Bootstrap starting.
I20260923 21:06:04.320568 21893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261: Neither blocks nor log segments found. Creating new log.
Sep 23 21:06:04 dist-test-slave-8xn5 krb5kdc[21813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 21:06:04.321650 21893 log.cc:824] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:04.325536 21893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261: No bootstrap required, opened a new log
I20260923 21:06:04.331708 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:04.308635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:49381 (local address 127.0.0.1:43507)
0923 21:06:04.309473 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:04.309489 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:04.309531 (+ 42us) server_negotiation.cc:407] Connection header received
0923 21:06:04.311319 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:04.311352 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:04.311803 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:04.312226 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:04.313741 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:04.315186 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:04.316557 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:04.317274 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:04.317471 (+ 197us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:04.321821 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:04.321832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:04.321852 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:04.321907 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:04.326164 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:04.327271 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:04.327280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:04.327291 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:04.327542 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:04.328040 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:04.328045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:04.328048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:04.328868 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:04.329206 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:04.329769 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 21:06:04.330155 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":354,"threads_started":1}
I20260923 21:06:04.336509 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:04.336463 21893 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441477e44e45d6aa3eef18f6d66261" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260923 21:06:04.336928 21893 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:04.337055 21893 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd441477e44e45d6aa3eef18f6d66261, State: Initialized, Role: FOLLOWER
I20260923 21:06:04.337711 21893 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441477e44e45d6aa3eef18f6d66261" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260923 21:06:04.337960 21893 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:04.338096 21893 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:04.338284 21893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:04.340817 21893 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441477e44e45d6aa3eef18f6d66261" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260923 21:06:04.341471 21893 leader_election.cc:302] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd441477e44e45d6aa3eef18f6d66261; no voters:
I20260923 21:06:04.342224 21893 leader_election.cc:288] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:04.342352 21903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:04.343770 21903 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [term 1 LEADER]: Becoming Leader. State: Replica: fd441477e44e45d6aa3eef18f6d66261, State: Running, Role: LEADER
I20260923 21:06:04.344524 21903 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441477e44e45d6aa3eef18f6d66261" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } }
I20260923 21:06:04.345098 21893 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:04.349412 21904 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd441477e44e45d6aa3eef18f6d66261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441477e44e45d6aa3eef18f6d66261" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } } }
I20260923 21:06:04.349866 21904 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:04.352326 21905 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd441477e44e45d6aa3eef18f6d66261. Latest consensus state: current_term: 1 leader_uuid: "fd441477e44e45d6aa3eef18f6d66261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd441477e44e45d6aa3eef18f6d66261" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43507 } } }
I20260923 21:06:04.352828 21905 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:04.354790 21908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:04.360731 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:04.341521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:37451 (local address 127.0.0.1:43507)
0923 21:06:04.341716 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:04.341724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:04.342221 (+ 497us) server_negotiation.cc:407] Connection header received
0923 21:06:04.342627 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:04.342635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:04.342750 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:04.342883 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:04.345255 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:04.346220 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:04.348416 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:04.349426 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:04.349617 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:04.353687 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:04.353695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:04.353706 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:04.353759 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:04.357077 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:04.357948 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:04.357953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:04.357957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:04.358109 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:04.358760 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:04.358767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:04.358772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:04.359257 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:04.359637 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:04.360059 (+ 422us) server_negotiation.cc:299] Negotiation successful
0923 21:06:04.360249 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 21:06:04.370186 21908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:04.379798 21908 catalog_manager.cc:1347] Generated new cluster ID: bfc71f567ba24710b6c9fd50f5d0dba9
I20260923 21:06:04.379938 21908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:04.394424 21908 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:04.395785 21908 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:04.404776 21908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd441477e44e45d6aa3eef18f6d66261: Generated new TSK 0
I20260923 21:06:04.405821 21908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:04.418015 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21826
I20260923 21:06:04.419431 21888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:04.516665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21826
W20260923 21:06:04.534729 21811 connection.cc:535] client connection to 127.0.0.1:43507 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43507): Connection reset by peer (error 104)
2026-09-23T21:06:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:04 dist-test-slave-8xn5 krb5kdc[21925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:04 dist-test-slave-8xn5 krb5kdc[21925](info): set up 2 sockets
Sep 23 21:06:04 dist-test-slave-8xn5 krb5kdc[21925](info): commencing operation
krb5kdc: starting...
W20260923 21:06:05.658967 388 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:05 dist-test-slave-8xn5 krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:06:05.934727 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:06.298291 21938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:06.298740 21938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:06.298905 21938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:06.307847 21938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:06.307983 21938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:06.308066 21938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:06.308163 21938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:06.308224 21938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:06.318717 21938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:06.320122 21938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:06.321578 21938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:06.335671 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:06.335556 21944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:06.335969 21943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:06.336530 21938 server_base.cc:1034] running on GCE node
I20260923 21:06:06.337451 21938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:06.339282 21938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:06.340554 21938 hybrid_clock.cc:648] HybridClock initialized: now 1790197566340512 us; error 35 us; skew 500 ppm
I20260923 21:06:06.341133 21938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:06 dist-test-slave-8xn5 krb5kdc[21925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:06.350688 21938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:06:06.352591 21938 webserver.cc:460] Webserver started at http://127.0.0.1:45731/ using document root <none> and password file <none>
I20260923 21:06:06.353396 21938 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:06.353519 21938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:06.353847 21938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:06.357450 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "9a88a79ee24044dfb95821bc2353f2b4"
format_stamp: "Formatted at 2026-09-23 21:06:06 on dist-test-slave-8xn5"
server_key: "b106392a6e401c332b3776f12791e015"
server_key_iv: "06e495a721adee78a072996ddbbeff84"
server_key_version: "encryptionkey@0"
I20260923 21:06:06.358309 21938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a88a79ee24044dfb95821bc2353f2b4"
format_stamp: "Formatted at 2026-09-23 21:06:06 on dist-test-slave-8xn5"
server_key: "b106392a6e401c332b3776f12791e015"
server_key_iv: "06e495a721adee78a072996ddbbeff84"
server_key_version: "encryptionkey@0"
I20260923 21:06:06.365195 21938 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 21:06:06.369398 21953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:06.371512 21938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260923 21:06:06.371783 21938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "9a88a79ee24044dfb95821bc2353f2b4"
format_stamp: "Formatted at 2026-09-23 21:06:06 on dist-test-slave-8xn5"
server_key: "b106392a6e401c332b3776f12791e015"
server_key_iv: "06e495a721adee78a072996ddbbeff84"
server_key_version: "encryptionkey@0"
I20260923 21:06:06.372020 21938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:06.395260 21938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:06.400044 21938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:06.400419 21938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:06.419749 21938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35483
I20260923 21:06:06.419782 22004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35483 every 8 connection(s)
I20260923 21:06:06.421221 21938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:06.422976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 21938
I20260923 21:06:06.423188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:06.423779 388 external_mini_cluster.cc:1428] Setting key 9b2c1300446a3619011d5cdb0dbbca3f
I20260923 21:06:06.426836 22005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:06.440135 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:06.425833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:33421 (local address 127.0.0.1:35483)
0923 21:06:06.426812 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:06.426830 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:06.426875 (+ 45us) server_negotiation.cc:407] Connection header received
0923 21:06:06.428768 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:06.428792 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:06.429226 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:06.429624 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:06.430446 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:06.432139 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:06.434480 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:06.435404 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:06.435587 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:06.437194 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:06.437207 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:06.437229 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:06.437929 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:06.438029 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:06.438069 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:06.438210 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:06.438694 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 21:06:06.438974 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":464,"threads_started":1}
I20260923 21:06:06.441263 22005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4: Bootstrap starting.
I20260923 21:06:06.444864 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:06.445497 22005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:06.446941 22005 log.cc:824] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:06.450237 22005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4: No bootstrap required, opened a new log
I20260923 21:06:06.459293 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:06.450798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:43749 (local address 127.0.0.1:35483)
0923 21:06:06.451013 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:06.451021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:06.451558 (+ 537us) server_negotiation.cc:407] Connection header received
0923 21:06:06.451784 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:06.451790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:06.451896 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:06.452039 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:06.452621 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:06.454030 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:06.455827 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:06.456745 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:06.456937 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:06.458236 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:06.458244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:06.458250 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:06.458279 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:06.458393 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:06.458428 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:06.458570 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:06.458762 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 21:06:06.458838 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:06:06.462016 22005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88a79ee24044dfb95821bc2353f2b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260923 21:06:06.462661 22005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:06.462798 22005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a88a79ee24044dfb95821bc2353f2b4, State: Initialized, Role: FOLLOWER
I20260923 21:06:06.463666 22005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88a79ee24044dfb95821bc2353f2b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260923 21:06:06.463955 22005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:06.464090 22005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:06.464262 22005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:06.467203 22005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88a79ee24044dfb95821bc2353f2b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260923 21:06:06.467845 22005 leader_election.cc:302] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a88a79ee24044dfb95821bc2353f2b4; no voters:
I20260923 21:06:06.468806 22005 leader_election.cc:288] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:06.468953 22015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:06.470520 22015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [term 1 LEADER]: Becoming Leader. State: Replica: 9a88a79ee24044dfb95821bc2353f2b4, State: Running, Role: LEADER
I20260923 21:06:06.471168 22015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88a79ee24044dfb95821bc2353f2b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } }
I20260923 21:06:06.471848 22005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:06.474457 22017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a88a79ee24044dfb95821bc2353f2b4. Latest consensus state: current_term: 1 leader_uuid: "9a88a79ee24044dfb95821bc2353f2b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88a79ee24044dfb95821bc2353f2b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } } }
I20260923 21:06:06.474921 22017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:06.475507 22016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a88a79ee24044dfb95821bc2353f2b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a88a79ee24044dfb95821bc2353f2b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35483 } } }
I20260923 21:06:06.475797 22016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:06.483227 22023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:06.489413 22023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:06.500555 22023 catalog_manager.cc:1347] Generated new cluster ID: 081dca866de742389dda52afb8d99bc2
I20260923 21:06:06.500727 22023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:06.523299 22023 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:06.524915 22023 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:06.533612 22023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a88a79ee24044dfb95821bc2353f2b4: Generated new TSK 0
I20260923 21:06:06.534709 22023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:06.541978 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21938
W20260923 21:06:06.543869 22000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:33421
W20260923 21:06:06.544678 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 21938: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:06.544837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 21938
W20260923 21:06:06.562470 21923 connection.cc:535] client connection to 127.0.0.1:35483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35483): Connection reset by peer (error 104)
2026-09-23T21:06:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:06 dist-test-slave-8xn5 krb5kdc[22036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:06 dist-test-slave-8xn5 krb5kdc[22036](info): set up 2 sockets
Sep 23 21:06:06 dist-test-slave-8xn5 krb5kdc[22036](info): commencing operation
krb5kdc: starting...
W20260923 21:06:07.671631 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:07 dist-test-slave-8xn5 krb5kdc[22036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 21:06:07.950654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:08.300452 22049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:08.300765 22049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:08.300942 22049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:08.308800 22049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:08.308933 22049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:08.309026 22049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:08.309120 22049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:08.309208 22049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:08.319749 22049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34337
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:08.321115 22049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:08.322603 22049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:08.335886 22054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:08.335901 22057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:08.335906 22055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:08.336728 22049 server_base.cc:1034] running on GCE node
I20260923 21:06:08.337641 22049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:08.339239 22049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:08.340525 22049 hybrid_clock.cc:648] HybridClock initialized: now 1790197568340449 us; error 72 us; skew 500 ppm
I20260923 21:06:08.340932 22049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:08 dist-test-slave-8xn5 krb5kdc[22036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:08.344462 22049 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 21:06:08.346114 22049 webserver.cc:460] Webserver started at http://127.0.0.1:41349/ using document root <none> and password file <none>
I20260923 21:06:08.346916 22049 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:08.347069 22049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:08.347383 22049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:08.351090 22049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "8cbd3a51170648ed8dc0dd7735adab4b"
format_stamp: "Formatted at 2026-09-23 21:06:08 on dist-test-slave-8xn5"
server_key: "530bb237ca0f3caf1725515bcaf3d97f"
server_key_iv: "3802a84f321eb7b708aa384ff2b1352e"
server_key_version: "encryptionkey@0"
I20260923 21:06:08.351975 22049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "8cbd3a51170648ed8dc0dd7735adab4b"
format_stamp: "Formatted at 2026-09-23 21:06:08 on dist-test-slave-8xn5"
server_key: "530bb237ca0f3caf1725515bcaf3d97f"
server_key_iv: "3802a84f321eb7b708aa384ff2b1352e"
server_key_version: "encryptionkey@0"
I20260923 21:06:08.357955 22049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260923 21:06:08.362020 22064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:08.363591 22049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:06:08.363894 22049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "8cbd3a51170648ed8dc0dd7735adab4b"
format_stamp: "Formatted at 2026-09-23 21:06:08 on dist-test-slave-8xn5"
server_key: "530bb237ca0f3caf1725515bcaf3d97f"
server_key_iv: "3802a84f321eb7b708aa384ff2b1352e"
server_key_version: "encryptionkey@0"
I20260923 21:06:08.364332 22049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:08.370450 22049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:08.375142 22049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:08.375708 22049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:08.397456 22049 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34337
I20260923 21:06:08.397509 22115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34337 every 8 connection(s)
I20260923 21:06:08.399114 22049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:08.403450 22116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:08.408491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22049
I20260923 21:06:08.408684 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:08.409261 388 external_mini_cluster.cc:1428] Setting key 7921981de02516853d0f7b71e0d9f355
I20260923 21:06:08.416659 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b: Bootstrap starting.
I20260923 21:06:08.418301 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:08.411350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:53537 (local address 127.0.0.1:34337)
0923 21:06:08.412217 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:08.412229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:08.412272 (+ 43us) server_negotiation.cc:407] Connection header received
0923 21:06:08.414726 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:08.414751 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:08.415117 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:08.415463 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:08.416609 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:08.416615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:08.416628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:08.416647 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:08.416705 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:08.416725 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:08.416961 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:08.417091 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 21:06:08.417355 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":438,"threads_started":1}
I20260923 21:06:08.420739 22116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:08.422075 22116 log.cc:824] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:08.422437 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:08.425362 22116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b: No bootstrap required, opened a new log
I20260923 21:06:08.429347 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:08.426692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.43:48295 (local address 127.0.0.1:34337)
0923 21:06:08.426880 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:08.426890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:08.427370 (+ 480us) server_negotiation.cc:407] Connection header received
0923 21:06:08.427485 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:08.427493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:08.427574 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:08.427707 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:08.428638 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:08.428645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:08.428650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:08.428671 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:08.428725 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:08.428747 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:08.428873 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:08.428928 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 21:06:08.428996 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 21:06:08.434988 22116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260923 21:06:08.435520 22116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:08.435618 22116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbd3a51170648ed8dc0dd7735adab4b, State: Initialized, Role: FOLLOWER
I20260923 21:06:08.436218 22116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260923 21:06:08.436452 22116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:08.436563 22116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:08.436721 22116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:08.439204 22116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260923 21:06:08.439846 22116 leader_election.cc:302] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cbd3a51170648ed8dc0dd7735adab4b; no voters:
I20260923 21:06:08.440771 22116 leader_election.cc:288] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:08.440908 22126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:08.441635 22126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [term 1 LEADER]: Becoming Leader. State: Replica: 8cbd3a51170648ed8dc0dd7735adab4b, State: Running, Role: LEADER
I20260923 21:06:08.442370 22126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } }
I20260923 21:06:08.443466 22116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:08.445513 22127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } } }
I20260923 21:06:08.445968 22127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:08.446498 22128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cbd3a51170648ed8dc0dd7735adab4b. Latest consensus state: current_term: 1 leader_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cbd3a51170648ed8dc0dd7735adab4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34337 } } }
I20260923 21:06:08.446890 22128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:08.448762 22134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:08.454458 22134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:08.464298 22134 catalog_manager.cc:1347] Generated new cluster ID: 8a3b86d90b984f748ed347c7d421c66a
I20260923 21:06:08.464433 22134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:08.477259 22134 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:08.477808 22134 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:08.488320 22134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cbd3a51170648ed8dc0dd7735adab4b: Generated new TSK 0
I20260923 21:06:08.489494 22134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:08.502992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22049
W20260923 21:06:08.504436 22111 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.43:53537
W20260923 21:06:08.505291 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22049: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:08.505452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22049
2026-09-23T21:06:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1957 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:08 dist-test-slave-8xn5 krb5kdc[22147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:08 dist-test-slave-8xn5 krb5kdc[22147](info): set up 2 sockets
Sep 23 21:06:08 dist-test-slave-8xn5 krb5kdc[22147](info): commencing operation
krb5kdc: starting...
W20260923 21:06:09.710843 388 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:09 dist-test-slave-8xn5 krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:09.983505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35441
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:10.334658 22160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:10.335002 22160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:10.335100 22160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:10.342963 22160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:10.343108 22160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:10.343155 22160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:10.343210 22160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:10.343255 22160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:10.353824 22160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:10.355238 22160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:10.356642 22160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:10.371372 22166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:10.371441 22168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:10.371443 22165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:10.372802 22160 server_base.cc:1034] running on GCE node
I20260923 21:06:10.373795 22160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:10.375689 22160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:10.376957 22160 hybrid_clock.cc:648] HybridClock initialized: now 1790197570376861 us; error 121 us; skew 500 ppm
I20260923 21:06:10.377480 22160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:10 dist-test-slave-8xn5 krb5kdc[22147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:10.381115 22160 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:10.383005 22160 webserver.cc:460] Webserver started at http://127.0.97.62:39855/ using document root <none> and password file <none>
I20260923 21:06:10.383809 22160 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:10.383927 22160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:10.384212 22160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:10.387651 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "ffa64e810a9a43f1b9fdf6b457647ee4"
format_stamp: "Formatted at 2026-09-23 21:06:10 on dist-test-slave-8xn5"
server_key: "b551813c05e85e52cf4b878435177c19"
server_key_iv: "174c56efb29515f11a28660c4c2e2293"
server_key_version: "encryptionkey@0"
I20260923 21:06:10.388473 22160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "ffa64e810a9a43f1b9fdf6b457647ee4"
format_stamp: "Formatted at 2026-09-23 21:06:10 on dist-test-slave-8xn5"
server_key: "b551813c05e85e52cf4b878435177c19"
server_key_iv: "174c56efb29515f11a28660c4c2e2293"
server_key_version: "encryptionkey@0"
I20260923 21:06:10.394876 22160 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:06:10.398875 22175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:10.400822 22160 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 21:06:10.400995 22160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "ffa64e810a9a43f1b9fdf6b457647ee4"
format_stamp: "Formatted at 2026-09-23 21:06:10 on dist-test-slave-8xn5"
server_key: "b551813c05e85e52cf4b878435177c19"
server_key_iv: "174c56efb29515f11a28660c4c2e2293"
server_key_version: "encryptionkey@0"
I20260923 21:06:10.401206 22160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:10.415808 22160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:10.420547 22160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:10.420899 22160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:10.441059 22160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35441
I20260923 21:06:10.441167 22226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35441 every 8 connection(s)
I20260923 21:06:10.442595 22160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:10.446784 22227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:10.451802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22160
I20260923 21:06:10.451977 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:10.452423 388 external_mini_cluster.cc:1428] Setting key 9f7bab162fc27478e561adae1f3d5633
I20260923 21:06:10.459407 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4: Bootstrap starting.
I20260923 21:06:10.462826 22227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:10.464035 22227 log.cc:824] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:06:10 dist-test-slave-8xn5 krb5kdc[22147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:06:10.467579 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4: No bootstrap required, opened a new log
I20260923 21:06:10.477717 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:10.454481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41297 (local address 127.0.97.62:35441)
0923 21:06:10.455461 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:10.455474 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:10.455513 (+ 39us) server_negotiation.cc:407] Connection header received
0923 21:06:10.457679 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:10.457704 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:10.458193 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:10.458669 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:10.460200 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:10.461541 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:10.463025 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:10.463763 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:10.463986 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:10.469072 (+ 5086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:10.469147 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:10.469165 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:10.469212 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:10.472628 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:10.473736 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:10.473747 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:10.473755 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:10.473956 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:10.474525 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:10.474529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:10.474531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:10.475179 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:10.475457 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:10.476005 (+ 548us) server_negotiation.cc:299] Negotiation successful
0923 21:06:10.476488 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":514,"threads_started":1}
I20260923 21:06:10.477619 22227 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35441 } }
I20260923 21:06:10.478132 22227 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:10.478289 22227 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa64e810a9a43f1b9fdf6b457647ee4, State: Initialized, Role: FOLLOWER
I20260923 21:06:10.478965 22227 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35441 } }
I20260923 21:06:10.479254 22227 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:10.479435 22227 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:10.479629 22227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:10.482159 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:10.482301 22227 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35441 } }
I20260923 21:06:10.482956 22227 leader_election.cc:302] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa64e810a9a43f1b9fdf6b457647ee4; no voters:
I20260923 21:06:10.483624 22227 leader_election.cc:288] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:10.483772 22232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:10.485276 22232 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [term 1 LEADER]: Becoming Leader. State: Replica: ffa64e810a9a43f1b9fdf6b457647ee4, State: Running, Role: LEADER
I20260923 21:06:10.485913 22232 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35441 } }
I20260923 21:06:10.486145 22227 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:10.489142 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa64e810a9a43f1b9fdf6b457647ee4. Latest consensus state: current_term: 1 leader_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35441 } } }
I20260923 21:06:10.489176 22233 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa64e810a9a43f1b9fdf6b457647ee4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35441 } } }
I20260923 21:06:10.489727 22233 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:10.490266 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:10.493516 22240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:10.499938 22240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:10.514154 22240 catalog_manager.cc:1347] Generated new cluster ID: 3f362f71584043bbb17d9b11d21da328
I20260923 21:06:10.514305 22240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:10.534101 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:10.495482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43649 (local address 127.0.97.62:35441)
0923 21:06:10.495707 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:10.495742 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:10.510576 (+ 14834us) server_negotiation.cc:407] Connection header received
0923 21:06:10.510908 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:10.510914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:10.511007 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:10.511087 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:10.513326 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:10.514575 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:10.520546 (+ 5971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:10.521379 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:10.521535 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:10.525051 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:10.525059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:10.525065 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:10.525113 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:10.529255 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:10.530535 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:10.530542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:10.530545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:10.530719 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:10.531392 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:10.531397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:10.531400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:10.531869 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:10.532160 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:10.533450 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0923 21:06:10.533682 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 21:06:10.537845 22240 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:10.539347 22240 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:10.548799 22240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffa64e810a9a43f1b9fdf6b457647ee4: Generated new TSK 0
I20260923 21:06:10.549715 22240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:10.558236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22160
I20260923 21:06:10.560114 22222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:10.657713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22160
W20260923 21:06:10.675406 22145 connection.cc:535] client connection to 127.0.97.62:35441 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35441): Connection reset by peer (error 104)
2026-09-23T21:06:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2155 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:10 dist-test-slave-8xn5 krb5kdc[22259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:10 dist-test-slave-8xn5 krb5kdc[22259](info): set up 2 sockets
Sep 23 21:06:10 dist-test-slave-8xn5 krb5kdc[22259](info): commencing operation
krb5kdc: starting...
W20260923 21:06:11.855520 388 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:12 dist-test-slave-8xn5 krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:12.139034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44573
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:12.507258 22272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:12.507670 22272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:12.507851 22272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:12.515712 22272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:12.515906 22272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:12.516036 22272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:12.516167 22272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:12.516268 22272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:12.528009 22272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:12.529392 22272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:12.530891 22272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:12.544375 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:12.544443 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:12.544342 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:12.545207 22272 server_base.cc:1034] running on GCE node
I20260923 21:06:12.546036 22272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:12.547681 22272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:12.549012 22272 hybrid_clock.cc:648] HybridClock initialized: now 1790197572548966 us; error 46 us; skew 500 ppm
I20260923 21:06:12.549433 22272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:12 dist-test-slave-8xn5 krb5kdc[22259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:12.553141 22272 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:12.554877 22272 webserver.cc:460] Webserver started at http://127.0.97.62:38829/ using document root <none> and password file <none>
I20260923 21:06:12.555600 22272 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:12.555708 22272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:12.556002 22272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:12.559334 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "fc2e0b4f20954190a19568c367197b59"
format_stamp: "Formatted at 2026-09-23 21:06:12 on dist-test-slave-8xn5"
server_key: "ef6c767e9c789b50569228cdeb3835c8"
server_key_iv: "90fa046ded2f763bee7fdf9710c0cf5d"
server_key_version: "encryptionkey@0"
I20260923 21:06:12.560128 22272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc2e0b4f20954190a19568c367197b59"
format_stamp: "Formatted at 2026-09-23 21:06:12 on dist-test-slave-8xn5"
server_key: "ef6c767e9c789b50569228cdeb3835c8"
server_key_iv: "90fa046ded2f763bee7fdf9710c0cf5d"
server_key_version: "encryptionkey@0"
I20260923 21:06:12.565982 22272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:06:12.570132 22287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:12.571846 22272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 21:06:12.572016 22272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "fc2e0b4f20954190a19568c367197b59"
format_stamp: "Formatted at 2026-09-23 21:06:12 on dist-test-slave-8xn5"
server_key: "ef6c767e9c789b50569228cdeb3835c8"
server_key_iv: "90fa046ded2f763bee7fdf9710c0cf5d"
server_key_version: "encryptionkey@0"
I20260923 21:06:12.572202 22272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:12.601562 22272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:12.607013 22272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:12.607275 22272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:12.627669 22272 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44573
I20260923 21:06:12.627710 22338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44573 every 8 connection(s)
I20260923 21:06:12.629119 22272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:12.633122 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:12.636646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22272
I20260923 21:06:12.636847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:12.637291 388 external_mini_cluster.cc:1428] Setting key c5465c54b652b17a7cb802e7c1121fe2
I20260923 21:06:12.646620 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59: Bootstrap starting.
I20260923 21:06:12.650799 22339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:12.651973 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:12.639279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40963 (local address 127.0.97.62:44573)
0923 21:06:12.640094 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:12.640106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:12.640149 (+ 43us) server_negotiation.cc:407] Connection header received
0923 21:06:12.641846 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:12.641868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:12.642390 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:12.642756 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:12.643403 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:12.644843 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:12.646882 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:12.647593 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:12.647729 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:12.649306 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:12.649316 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:12.649332 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:12.650000 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:12.650090 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:12.650121 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:12.650276 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:12.650659 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 21:06:12.650857 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":387,"threads_started":1}
I20260923 21:06:12.652056 22339 log.cc:824] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:12.655323 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59: No bootstrap required, opened a new log
I20260923 21:06:12.656543 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:12.664216 22339 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0b4f20954190a19568c367197b59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44573 } }
I20260923 21:06:12.664773 22339 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:12.664893 22339 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2e0b4f20954190a19568c367197b59, State: Initialized, Role: FOLLOWER
I20260923 21:06:12.665491 22339 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0b4f20954190a19568c367197b59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44573 } }
I20260923 21:06:12.665732 22339 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:12.665895 22339 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:12.666083 22339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:12.669215 22339 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0b4f20954190a19568c367197b59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44573 } }
I20260923 21:06:12.669425 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:12.661520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55695 (local address 127.0.97.62:44573)
0923 21:06:12.661744 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:12.661750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:12.662275 (+ 525us) server_negotiation.cc:407] Connection header received
0923 21:06:12.662536 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:12.662543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:12.662627 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:12.662755 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:12.663347 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:12.664410 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:12.666383 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:12.667094 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:12.667239 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:12.668504 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:12.668510 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:12.668515 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:12.668537 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:12.668583 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:12.668602 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:12.668724 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:12.668960 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 21:06:12.669021 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:06:12.669832 22339 leader_election.cc:302] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc2e0b4f20954190a19568c367197b59; no voters:
I20260923 21:06:12.670554 22339 leader_election.cc:288] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:12.670730 22349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:12.672492 22339 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:12.677702 22349 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [term 1 LEADER]: Becoming Leader. State: Replica: fc2e0b4f20954190a19568c367197b59, State: Running, Role: LEADER
I20260923 21:06:12.678262 22349 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0b4f20954190a19568c367197b59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44573 } }
I20260923 21:06:12.684312 22351 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc2e0b4f20954190a19568c367197b59. Latest consensus state: current_term: 1 leader_uuid: "fc2e0b4f20954190a19568c367197b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0b4f20954190a19568c367197b59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44573 } } }
I20260923 21:06:12.684765 22351 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:12.684715 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc2e0b4f20954190a19568c367197b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2e0b4f20954190a19568c367197b59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44573 } } }
I20260923 21:06:12.685009 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:12.687181 22362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:12.692231 22362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:12.702383 22362 catalog_manager.cc:1347] Generated new cluster ID: 7bb73dc0106245f4b138a27a6c6b1bfa
I20260923 21:06:12.702524 22362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:12.718082 22362 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:12.719535 22362 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:12.727089 22362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc2e0b4f20954190a19568c367197b59: Generated new TSK 0
I20260923 21:06:12.728027 22362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:12.738643 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22272
W20260923 21:06:12.740298 22334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40963
W20260923 21:06:12.741241 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22272: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:12.741412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22272
W20260923 21:06:12.759308 22257 connection.cc:535] client connection to 127.0.97.62:44573 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44573): Connection reset by peer (error 104)
2026-09-23T21:06:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2083 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:12 dist-test-slave-8xn5 krb5kdc[22370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:12 dist-test-slave-8xn5 krb5kdc[22370](info): set up 2 sockets
Sep 23 21:06:12 dist-test-slave-8xn5 krb5kdc[22370](info): commencing operation
krb5kdc: starting...
W20260923 21:06:13.874779 388 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:14 dist-test-slave-8xn5 krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:14.161726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38207
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:14.527676 22383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:14.528149 22383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:14.528331 22383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:14.536079 22383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:14.536247 22383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:14.536353 22383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:14.536446 22383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:14.536544 22383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:14.547915 22383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:14.549443 22383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:14.550920 22383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:14.563472 22388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:14.563369 22391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:14.563519 22389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:14.564268 22383 server_base.cc:1034] running on GCE node
I20260923 21:06:14.565135 22383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:14.566744 22383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:14.568037 22383 hybrid_clock.cc:648] HybridClock initialized: now 1790197574567992 us; error 36 us; skew 500 ppm
I20260923 21:06:14.568440 22383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:14 dist-test-slave-8xn5 krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:14.572134 22383 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:14.573833 22383 webserver.cc:460] Webserver started at http://127.0.97.62:39179/ using document root <none> and password file <none>
I20260923 21:06:14.574690 22383 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:14.574841 22383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:14.575155 22383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:14.578289 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "dbcf52c89a8f46d7b8569ca9302a876a"
format_stamp: "Formatted at 2026-09-23 21:06:14 on dist-test-slave-8xn5"
server_key: "d1455a668b972dbec5ee352bffbcaffc"
server_key_iv: "6f2c263819bac2b3b639778df352a65c"
server_key_version: "encryptionkey@0"
I20260923 21:06:14.579176 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "dbcf52c89a8f46d7b8569ca9302a876a"
format_stamp: "Formatted at 2026-09-23 21:06:14 on dist-test-slave-8xn5"
server_key: "d1455a668b972dbec5ee352bffbcaffc"
server_key_iv: "6f2c263819bac2b3b639778df352a65c"
server_key_version: "encryptionkey@0"
I20260923 21:06:14.584724 22383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 21:06:14.588842 22398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:14.590652 22383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 21:06:14.590898 22383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "dbcf52c89a8f46d7b8569ca9302a876a"
format_stamp: "Formatted at 2026-09-23 21:06:14 on dist-test-slave-8xn5"
server_key: "d1455a668b972dbec5ee352bffbcaffc"
server_key_iv: "6f2c263819bac2b3b639778df352a65c"
server_key_version: "encryptionkey@0"
I20260923 21:06:14.591156 22383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:14.598160 22383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:14.602866 22383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:14.603196 22383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:14.624076 22383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38207
I20260923 21:06:14.624116 22449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38207 every 8 connection(s)
I20260923 21:06:14.625600 22383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:14.629248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22383
I20260923 21:06:14.629441 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:14.629985 388 external_mini_cluster.cc:1428] Setting key fb6f704ca1bd0794efc41f01d59685d6
I20260923 21:06:14.630038 22450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:14.639300 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:14.632030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54159 (local address 127.0.97.62:38207)
0923 21:06:14.632928 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:14.632940 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:14.632984 (+ 44us) server_negotiation.cc:407] Connection header received
0923 21:06:14.635302 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:14.635330 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:14.635761 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:14.636195 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:14.637342 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:14.637349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:14.637368 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:14.637389 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:14.637456 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:14.637490 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:14.637709 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:14.637836 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 21:06:14.638104 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":438,"threads_started":1}
I20260923 21:06:14.643883 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:14.644564 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a: Bootstrap starting.
I20260923 21:06:14.648732 22450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:14.650007 22450 log.cc:824] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:14.652722 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:14.649476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48637 (local address 127.0.97.62:38207)
0923 21:06:14.649681 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:14.649688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:14.650286 (+ 598us) server_negotiation.cc:407] Connection header received
0923 21:06:14.650509 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:14.650516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:14.650589 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:14.650732 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:14.651787 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:14.651793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:14.651798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:14.651985 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:14.652033 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:14.652066 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:14.652203 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:14.652349 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 21:06:14.652420 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:06:14.653894 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a: No bootstrap required, opened a new log
I20260923 21:06:14.661558 22450 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38207 } }
I20260923 21:06:14.661949 22450 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:14.662016 22450 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcf52c89a8f46d7b8569ca9302a876a, State: Initialized, Role: FOLLOWER
I20260923 21:06:14.662731 22450 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38207 } }
I20260923 21:06:14.662942 22450 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:14.663117 22450 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:14.663282 22450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:14.665485 22450 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38207 } }
I20260923 21:06:14.666035 22450 leader_election.cc:302] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbcf52c89a8f46d7b8569ca9302a876a; no voters:
I20260923 21:06:14.666814 22450 leader_election.cc:288] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:14.666970 22460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:14.667733 22460 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [term 1 LEADER]: Becoming Leader. State: Replica: dbcf52c89a8f46d7b8569ca9302a876a, State: Running, Role: LEADER
I20260923 21:06:14.668272 22460 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38207 } }
I20260923 21:06:14.668995 22450 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:14.671275 22461 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38207 } } }
I20260923 21:06:14.671756 22461 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:14.672539 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcf52c89a8f46d7b8569ca9302a876a. Latest consensus state: current_term: 1 leader_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf52c89a8f46d7b8569ca9302a876a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38207 } } }
I20260923 21:06:14.672887 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:14.674443 22469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:14.679880 22469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:14.689215 22469 catalog_manager.cc:1347] Generated new cluster ID: 67340c4bdb65451f918aa6fae101c022
I20260923 21:06:14.689328 22469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:14.711699 22469 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:14.712178 22469 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:14.719188 22469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbcf52c89a8f46d7b8569ca9302a876a: Generated new TSK 0
I20260923 21:06:14.720165 22469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:14.729012 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22383
W20260923 21:06:14.730710 22445 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54159
W20260923 21:06:14.731578 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22383: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:14.731716 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22383
2026-09-23T21:06:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:14 dist-test-slave-8xn5 krb5kdc[22481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:14 dist-test-slave-8xn5 krb5kdc[22481](info): set up 2 sockets
Sep 23 21:06:14 dist-test-slave-8xn5 krb5kdc[22481](info): commencing operation
krb5kdc: starting...
W20260923 21:06:15.891208 388 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:16 dist-test-slave-8xn5 krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:16.170442 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38535
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:16.516340 22494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:16.516746 22494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:16.516917 22494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:16.524621 22494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:16.524803 22494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:16.524923 22494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:16.525041 22494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:16.525120 22494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:16.536966 22494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:16.538672 22494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:16.540498 22494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:16.554353 22500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:16.554407 22502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:16.554432 22499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:16.555284 22494 server_base.cc:1034] running on GCE node
I20260923 21:06:16.556196 22494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:16.557832 22494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:16.559098 22494 hybrid_clock.cc:648] HybridClock initialized: now 1790197576559021 us; error 59 us; skew 500 ppm
I20260923 21:06:16.559514 22494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:16 dist-test-slave-8xn5 krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:16.563052 22494 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:16.564613 22494 webserver.cc:460] Webserver started at http://127.0.97.62:42331/ using document root <none> and password file <none>
I20260923 21:06:16.565320 22494 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:16.565433 22494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:16.565706 22494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:16.569046 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "b54b5098fe1843beb9b927802e46d27c"
format_stamp: "Formatted at 2026-09-23 21:06:16 on dist-test-slave-8xn5"
server_key: "c99c12ae59cf00f205a55f12041def93"
server_key_iv: "92fee3a2717262ab770682d7cdea0c47"
server_key_version: "encryptionkey@0"
I20260923 21:06:16.569895 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "b54b5098fe1843beb9b927802e46d27c"
format_stamp: "Formatted at 2026-09-23 21:06:16 on dist-test-slave-8xn5"
server_key: "c99c12ae59cf00f205a55f12041def93"
server_key_iv: "92fee3a2717262ab770682d7cdea0c47"
server_key_version: "encryptionkey@0"
I20260923 21:06:16.575662 22494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 21:06:16.579509 22509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:16.581056 22494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:06:16.581274 22494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "b54b5098fe1843beb9b927802e46d27c"
format_stamp: "Formatted at 2026-09-23 21:06:16 on dist-test-slave-8xn5"
server_key: "c99c12ae59cf00f205a55f12041def93"
server_key_iv: "92fee3a2717262ab770682d7cdea0c47"
server_key_version: "encryptionkey@0"
I20260923 21:06:16.581496 22494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:16.611850 22494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:16.616983 22494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:16.617349 22494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:16.637276 22494 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38535
I20260923 21:06:16.637346 22560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38535 every 8 connection(s)
I20260923 21:06:16.638787 22494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:16.642910 22561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:16.647943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22494
I20260923 21:06:16.648150 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:16.648684 388 external_mini_cluster.cc:1428] Setting key e3b6388473e52ad82f8f75382e37c5b9
I20260923 21:06:16.654886 22561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c: Bootstrap starting.
I20260923 21:06:16.658814 22561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:16.660128 22561 log.cc:824] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:16.663602 22561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c: No bootstrap required, opened a new log
Sep 23 21:06:16 dist-test-slave-8xn5 krb5kdc[22481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:06:16.674016 22561 raft_consensus.cc:348] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54b5098fe1843beb9b927802e46d27c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38535 } }
I20260923 21:06:16.674611 22561 raft_consensus.cc:374] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:16.674765 22561 raft_consensus.cc:729] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54b5098fe1843beb9b927802e46d27c, State: Initialized, Role: FOLLOWER
I20260923 21:06:16.675014 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:16.650736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33441 (local address 127.0.97.62:38535)
0923 21:06:16.651662 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:16.651674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:16.651724 (+ 50us) server_negotiation.cc:407] Connection header received
0923 21:06:16.653586 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:16.653611 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:16.654099 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:16.654541 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:16.656424 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:16.657931 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:16.659866 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:16.661064 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:16.661319 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:16.665994 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:16.666004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:16.666023 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:16.666072 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:16.669986 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:16.671009 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:16.671017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:16.671025 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:16.671205 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:16.671699 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:16.671702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:16.671704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:16.672300 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:16.672599 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:16.673135 (+ 536us) server_negotiation.cc:299] Negotiation successful
0923 21:06:16.673609 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":392,"threads_started":1}
I20260923 21:06:16.675542 22561 consensus_queue.cc:260] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54b5098fe1843beb9b927802e46d27c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38535 } }
I20260923 21:06:16.675856 22561 raft_consensus.cc:388] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:16.676017 22561 raft_consensus.cc:482] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:16.676241 22561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:16.679615 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:16.679813 22561 raft_consensus.cc:504] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54b5098fe1843beb9b927802e46d27c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38535 } }
I20260923 21:06:16.680493 22561 leader_election.cc:302] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b54b5098fe1843beb9b927802e46d27c; no voters:
I20260923 21:06:16.681211 22561 leader_election.cc:288] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:16.681401 22566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:16.682914 22566 raft_consensus.cc:686] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [term 1 LEADER]: Becoming Leader. State: Replica: b54b5098fe1843beb9b927802e46d27c, State: Running, Role: LEADER
I20260923 21:06:16.683568 22566 consensus_queue.cc:237] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54b5098fe1843beb9b927802e46d27c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38535 } }
I20260923 21:06:16.684078 22561 sys_catalog.cc:564] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:16.686527 22568 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b54b5098fe1843beb9b927802e46d27c. Latest consensus state: current_term: 1 leader_uuid: "b54b5098fe1843beb9b927802e46d27c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54b5098fe1843beb9b927802e46d27c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38535 } } }
I20260923 21:06:16.686910 22568 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:16.687487 22567 sys_catalog.cc:455] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b54b5098fe1843beb9b927802e46d27c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b54b5098fe1843beb9b927802e46d27c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38535 } } }
I20260923 21:06:16.687794 22567 sys_catalog.cc:458] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:16.690557 22573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:16.696733 22573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:16.717386 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:16.692253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37407 (local address 127.0.97.62:38535)
0923 21:06:16.692515 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:16.692525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:16.693267 (+ 742us) server_negotiation.cc:407] Connection header received
0923 21:06:16.693661 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:16.693671 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:16.693789 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:16.693897 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:16.697226 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:16.698992 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:16.701921 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:16.702851 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:16.703002 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:16.707175 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:16.707186 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:16.707194 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:16.707261 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:16.712087 (+ 4826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:16.713285 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:16.713295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:16.713301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:16.713492 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:16.714021 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:16.714029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:16.714035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:16.715080 (+ 1045us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:16.715426 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:16.716348 (+ 922us) server_negotiation.cc:299] Negotiation successful
0923 21:06:16.716761 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 21:06:16.723053 22573 catalog_manager.cc:1347] Generated new cluster ID: 87ae89233eb04bdcbe1548b1ffa8a120
I20260923 21:06:16.723182 22573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:16.743046 22573 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:16.744504 22573 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:16.753948 22573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b54b5098fe1843beb9b927802e46d27c: Generated new TSK 0
I20260923 21:06:16.755203 22573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:16.770792 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22494
I20260923 21:06:16.771998 22556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:16.866797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22494
2026-09-23T21:06:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2135 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:16 dist-test-slave-8xn5 krb5kdc[22593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:16 dist-test-slave-8xn5 krb5kdc[22593](info): set up 2 sockets
Sep 23 21:06:16 dist-test-slave-8xn5 krb5kdc[22593](info): commencing operation
krb5kdc: starting...
W20260923 21:06:18.019027 388 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:18 dist-test-slave-8xn5 krb5kdc[22593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:18.295096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35361
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:18.652112 22606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:18.652410 22606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:18.652503 22606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:18.659794 22606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:18.659912 22606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:18.659958 22606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:18.660018 22606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:18.660063 22606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:18.670166 22606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:18.671579 22606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:18.672993 22606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:18.685494 22611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:18.685436 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:18.685405 22612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:18.686514 22606 server_base.cc:1034] running on GCE node
I20260923 21:06:18.687323 22606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:18.688812 22606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:18.690099 22606 hybrid_clock.cc:648] HybridClock initialized: now 1790197578690018 us; error 60 us; skew 500 ppm
I20260923 21:06:18.690603 22606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:18 dist-test-slave-8xn5 krb5kdc[22593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:18.700551 22606 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:18.702697 22606 webserver.cc:460] Webserver started at http://127.0.97.62:35799/ using document root <none> and password file <none>
I20260923 21:06:18.703452 22606 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:18.703564 22606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:18.703860 22606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:18.707248 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "da4bb2510a0240ed91a41e7ffbfd1be5"
format_stamp: "Formatted at 2026-09-23 21:06:18 on dist-test-slave-8xn5"
server_key: "dbc58439129dcb62712d24abb847b839"
server_key_iv: "e10d16bb912d97530c9f6cb4d23b5733"
server_key_version: "encryptionkey@0"
I20260923 21:06:18.708073 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "da4bb2510a0240ed91a41e7ffbfd1be5"
format_stamp: "Formatted at 2026-09-23 21:06:18 on dist-test-slave-8xn5"
server_key: "dbc58439129dcb62712d24abb847b839"
server_key_iv: "e10d16bb912d97530c9f6cb4d23b5733"
server_key_version: "encryptionkey@0"
I20260923 21:06:18.714624 22606 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:06:18.718778 22621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:18.720427 22606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:06:18.720641 22606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "da4bb2510a0240ed91a41e7ffbfd1be5"
format_stamp: "Formatted at 2026-09-23 21:06:18 on dist-test-slave-8xn5"
server_key: "dbc58439129dcb62712d24abb847b839"
server_key_iv: "e10d16bb912d97530c9f6cb4d23b5733"
server_key_version: "encryptionkey@0"
I20260923 21:06:18.720839 22606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:18.736146 22606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:18.740976 22606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:18.741307 22606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:18.762202 22606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35361
I20260923 21:06:18.762224 22672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35361 every 8 connection(s)
I20260923 21:06:18.763814 22606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:18.767817 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:18.773432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22606
I20260923 21:06:18.773607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:18.774129 388 external_mini_cluster.cc:1428] Setting key f1efae1338b7e1485b070e81926d9213
I20260923 21:06:18.779671 22673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5: Bootstrap starting.
I20260923 21:06:18.783485 22673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:18.784714 22673 log.cc:824] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:18.787716 22673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5: No bootstrap required, opened a new log
I20260923 21:06:18.788691 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:18.776183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47091 (local address 127.0.97.62:35361)
0923 21:06:18.777136 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:18.777149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:18.777201 (+ 52us) server_negotiation.cc:407] Connection header received
0923 21:06:18.779147 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:18.779171 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:18.779614 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:18.780041 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:18.780796 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:18.782161 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:18.784046 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:18.784836 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:18.785091 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:18.786304 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:18.786311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:18.786379 (+ 68us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:18.786951 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:18.787023 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:18.787054 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:18.787191 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:18.787409 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 21:06:18.787675 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":447,"threads_started":1}
I20260923 21:06:18.792505 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:18.796327 22673 raft_consensus.cc:348] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35361 } }
I20260923 21:06:18.796729 22673 raft_consensus.cc:374] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:18.796864 22673 raft_consensus.cc:729] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4bb2510a0240ed91a41e7ffbfd1be5, State: Initialized, Role: FOLLOWER
I20260923 21:06:18.797662 22673 consensus_queue.cc:260] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35361 } }
I20260923 21:06:18.797981 22673 raft_consensus.cc:388] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:18.798166 22673 raft_consensus.cc:482] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:18.798421 22673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:18.800930 22673 raft_consensus.cc:504] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35361 } }
I20260923 21:06:18.801461 22673 leader_election.cc:302] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da4bb2510a0240ed91a41e7ffbfd1be5; no voters:
I20260923 21:06:18.802189 22673 leader_election.cc:288] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:18.802372 22683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:18.803941 22683 raft_consensus.cc:686] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [term 1 LEADER]: Becoming Leader. State: Replica: da4bb2510a0240ed91a41e7ffbfd1be5, State: Running, Role: LEADER
I20260923 21:06:18.804636 22683 consensus_queue.cc:237] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35361 } }
I20260923 21:06:18.805167 22673 sys_catalog.cc:564] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:18.805348 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:18.797390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55527 (local address 127.0.97.62:35361)
0923 21:06:18.797667 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:18.797677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:18.798114 (+ 437us) server_negotiation.cc:407] Connection header received
0923 21:06:18.798372 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:18.798378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:18.798463 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:18.798596 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:18.799245 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:18.800067 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:18.801731 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:18.802616 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:18.802798 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:18.804162 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:18.804169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:18.804176 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:18.804203 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:18.804248 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:18.804270 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:18.804376 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:18.804879 (+ 503us) server_negotiation.cc:299] Negotiation successful
0923 21:06:18.804969 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 21:06:18.807493 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da4bb2510a0240ed91a41e7ffbfd1be5. Latest consensus state: current_term: 1 leader_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35361 } } }
I20260923 21:06:18.807823 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:18.808406 22684 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4bb2510a0240ed91a41e7ffbfd1be5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35361 } } }
I20260923 21:06:18.808776 22684 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:18.810376 22688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:18.816983 22688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:18.828241 22688 catalog_manager.cc:1347] Generated new cluster ID: 767f53ea6e03499b9e5c621472d27e57
I20260923 21:06:18.828390 22688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:18.856492 22688 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:18.858588 22688 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:18.870201 22688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da4bb2510a0240ed91a41e7ffbfd1be5: Generated new TSK 0
I20260923 21:06:18.871351 22688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:18.890389 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22606
W20260923 21:06:18.892282 22668 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47091
W20260923 21:06:18.893039 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22606: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:18.893220 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22606
2026-09-23T21:06:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:18 dist-test-slave-8xn5 krb5kdc[22704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:18 dist-test-slave-8xn5 krb5kdc[22704](info): set up 2 sockets
Sep 23 21:06:18 dist-test-slave-8xn5 krb5kdc[22704](info): commencing operation
krb5kdc: starting...
W20260923 21:06:20.047405 388 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:20 dist-test-slave-8xn5 krb5kdc[22704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:20.334613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44385
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:20.683247 22717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:20.683565 22717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:20.683801 22717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:20.691661 22717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:20.691795 22717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:20.691928 22717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:20.692040 22717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:20.692114 22717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:20.702659 22717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:20.704087 22717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:20.705528 22717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:20.718780 22722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:20.718842 22725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:20.718758 22723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:20.719949 22717 server_base.cc:1034] running on GCE node
I20260923 21:06:20.720897 22717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:20.722572 22717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:20.723855 22717 hybrid_clock.cc:648] HybridClock initialized: now 1790197580723792 us; error 68 us; skew 500 ppm
I20260923 21:06:20.724268 22717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:20 dist-test-slave-8xn5 krb5kdc[22704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:20.727941 22717 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:20.729724 22717 webserver.cc:460] Webserver started at http://127.0.97.62:36355/ using document root <none> and password file <none>
I20260923 21:06:20.730818 22717 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:20.730983 22717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:20.731351 22717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:20.734793 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "39d6ff2b27204921ae7488fdff27f933"
format_stamp: "Formatted at 2026-09-23 21:06:20 on dist-test-slave-8xn5"
server_key: "34b0b754e2128a5c00b17cb437ffce1f"
server_key_iv: "44d62d5dc7d6dd452a6621a4d8cdc7b3"
server_key_version: "encryptionkey@0"
I20260923 21:06:20.735651 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "39d6ff2b27204921ae7488fdff27f933"
format_stamp: "Formatted at 2026-09-23 21:06:20 on dist-test-slave-8xn5"
server_key: "34b0b754e2128a5c00b17cb437ffce1f"
server_key_iv: "44d62d5dc7d6dd452a6621a4d8cdc7b3"
server_key_version: "encryptionkey@0"
I20260923 21:06:20.741509 22717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 21:06:20.745267 22732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:20.746738 22717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 21:06:20.746938 22717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "39d6ff2b27204921ae7488fdff27f933"
format_stamp: "Formatted at 2026-09-23 21:06:20 on dist-test-slave-8xn5"
server_key: "34b0b754e2128a5c00b17cb437ffce1f"
server_key_iv: "44d62d5dc7d6dd452a6621a4d8cdc7b3"
server_key_version: "encryptionkey@0"
I20260923 21:06:20.747145 22717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:20.752916 22717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:20.757346 22717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:20.757665 22717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:20.777160 22717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44385
I20260923 21:06:20.777192 22783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44385 every 8 connection(s)
I20260923 21:06:20.778713 22717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:20.781329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22717
I20260923 21:06:20.781514 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:20.781971 388 external_mini_cluster.cc:1428] Setting key 1e9a9d7ec838a0762a9b569e1dd5e435
I20260923 21:06:20.783147 22784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:20.789883 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:20.783924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56795 (local address 127.0.97.62:44385)
0923 21:06:20.784713 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:20.784726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:20.784775 (+ 49us) server_negotiation.cc:407] Connection header received
0923 21:06:20.786373 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:20.786403 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:20.786789 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:20.787208 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:20.788249 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:20.788260 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:20.788275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:20.788289 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:20.788357 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:20.788382 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:20.788625 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:20.788778 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 21:06:20.788968 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":366,"threads_started":1}
I20260923 21:06:20.794207 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:20.796353 22784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933: Bootstrap starting.
I20260923 21:06:20.799741 22784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:20.801177 22784 log.cc:824] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:20.803318 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:20.800672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57767 (local address 127.0.97.62:44385)
0923 21:06:20.800913 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:20.800920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:20.801261 (+ 341us) server_negotiation.cc:407] Connection header received
0923 21:06:20.801475 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:20.801479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:20.801532 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:20.801641 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:20.802640 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:20.802646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:20.802650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 21:06:20.802668 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:20.802715 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 21:06:20.802730 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:20.802850 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:20.802983 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 21:06:20.803045 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:06:20.804457 22784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933: No bootstrap required, opened a new log
I20260923 21:06:20.812233 22784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d6ff2b27204921ae7488fdff27f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44385 } }
I20260923 21:06:20.812637 22784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 21:06:20.812748 22784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39d6ff2b27204921ae7488fdff27f933, State: Initialized, Role: FOLLOWER
I20260923 21:06:20.813405 22784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d6ff2b27204921ae7488fdff27f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44385 } }
I20260923 21:06:20.813606 22784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 21:06:20.813699 22784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 21:06:20.813915 22784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:20.816334 22784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d6ff2b27204921ae7488fdff27f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44385 } }
I20260923 21:06:20.816932 22784 leader_election.cc:302] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39d6ff2b27204921ae7488fdff27f933; no voters:
I20260923 21:06:20.817580 22784 leader_election.cc:288] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 21:06:20.817843 22794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:20.818809 22794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [term 1 LEADER]: Becoming Leader. State: Replica: 39d6ff2b27204921ae7488fdff27f933, State: Running, Role: LEADER
I20260923 21:06:20.819358 22794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d6ff2b27204921ae7488fdff27f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44385 } }
I20260923 21:06:20.819772 22784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:20.822736 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39d6ff2b27204921ae7488fdff27f933. Latest consensus state: current_term: 1 leader_uuid: "39d6ff2b27204921ae7488fdff27f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d6ff2b27204921ae7488fdff27f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44385 } } }
I20260923 21:06:20.823194 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:20.823124 22795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39d6ff2b27204921ae7488fdff27f933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39d6ff2b27204921ae7488fdff27f933" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44385 } } }
I20260923 21:06:20.823437 22795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:20.825402 22803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:20.830837 22803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:20.840231 22803 catalog_manager.cc:1347] Generated new cluster ID: 93837ddb78b645b9976766cb6b39b4cd
I20260923 21:06:20.840365 22803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:20.868680 22803 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:20.869153 22803 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:20.878860 22803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39d6ff2b27204921ae7488fdff27f933: Generated new TSK 0
I20260923 21:06:20.879724 22803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:20.896138 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22717
W20260923 21:06:20.897526 22779 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56795
W20260923 21:06:20.898304 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 21:06:20.898524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22717
2026-09-23T21:06:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2008 ms)
[----------] 18 tests from AuthTokenIssuingTest (36942 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:21 dist-test-slave-8xn5 krb5kdc[22815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:21 dist-test-slave-8xn5 krb5kdc[22815](info): set up 2 sockets
Sep 23 21:06:21 dist-test-slave-8xn5 krb5kdc[22815](info): commencing operation
krb5kdc: starting...
W20260923 21:06:22.063196 388 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:22 dist-test-slave-8xn5 krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:22.348136 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45209,127.0.97.61:39853,127.0.97.60:35159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:22.716316 22828 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 21:06:22.716691 22828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:22.716822 22828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:22.716982 22828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:22.724413 22828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:22.724510 22828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:22.724618 22828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:22.724715 22828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:22.724810 22828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:22.734994 22828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45209,127.0.97.61:39853,127.0.97.60:35159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:22.736392 22828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:22.737759 22828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:22.750310 22833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:22.750429 22836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:22.750439 22834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:22.751127 22828 server_base.cc:1034] running on GCE node
I20260923 21:06:22.751928 22828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:22.753434 22828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:22.754705 22828 hybrid_clock.cc:648] HybridClock initialized: now 1790197582754615 us; error 47 us; skew 500 ppm
I20260923 21:06:22.755160 22828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:22 dist-test-slave-8xn5 krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:22.759194 22828 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:22.760888 22828 webserver.cc:460] Webserver started at http://127.0.97.62:33947/ using document root <none> and password file <none>
I20260923 21:06:22.761739 22828 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:22.761902 22828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:22.762321 22828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:22.766008 22828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "396bd29b7ed74232a0798ea809ae41e3"
format_stamp: "Formatted at 2026-09-23 21:06:22 on dist-test-slave-8xn5"
server_key: "a887af0e050df1168d5f88a938bdae52"
server_key_iv: "7c54303d6fa036df5395725bdcd139e9"
server_key_version: "encryptionkey@0"
I20260923 21:06:22.767204 22828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "396bd29b7ed74232a0798ea809ae41e3"
format_stamp: "Formatted at 2026-09-23 21:06:22 on dist-test-slave-8xn5"
server_key: "a887af0e050df1168d5f88a938bdae52"
server_key_iv: "7c54303d6fa036df5395725bdcd139e9"
server_key_version: "encryptionkey@0"
I20260923 21:06:22.774304 22828 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260923 21:06:22.778290 22843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:22.779978 22828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 21:06:22.780247 22828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "396bd29b7ed74232a0798ea809ae41e3"
format_stamp: "Formatted at 2026-09-23 21:06:22 on dist-test-slave-8xn5"
server_key: "a887af0e050df1168d5f88a938bdae52"
server_key_iv: "7c54303d6fa036df5395725bdcd139e9"
server_key_version: "encryptionkey@0"
I20260923 21:06:22.780514 22828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:22.802733 22828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:22.807543 22828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:22.807873 22828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:22.827936 22828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45209
I20260923 21:06:22.827950 22894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45209 every 8 connection(s)
I20260923 21:06:22.829442 22828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:22.833339 22895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:22.836653 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22828
I20260923 21:06:22.836891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:22.837674 388 external_mini_cluster.cc:1428] Setting key 82ad85242f27db3ca775a28312978478
I20260923 21:06:22.842295 22895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:22 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260923 21:06:22.866453 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.846537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45209 (local address 127.0.0.1:59374)
0923 21:06:22.847558 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:22.847626 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:22.849627 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:22.850548 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:22.850554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:22.851294 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:22.851626 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:22.851644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:22.853226 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:22.853236 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:22.854378 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:22.854400 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:22.855339 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:22.855346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:22.855567 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:22.856258 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:22.856281 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:22.858525 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:22.861921 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:22.861931 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:22.861944 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:22.862406 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:22.862819 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:22.862826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:22.862829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:22.863024 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:22.863789 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:22.863797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:22.864057 (+ 260us) client_negotiation.cc:769] Sending connection context
0923 21:06:22.864356 (+ 299us) client_negotiation.cc:240] Negotiation successful
0923 21:06:22.864988 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":149,"thread_start_us":448,"threads_started":1}
I20260923 21:06:22.866472 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.846537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.97.62:45209)
0923 21:06:22.848214 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:22.848228 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:22.848276 (+ 48us) server_negotiation.cc:407] Connection header received
0923 21:06:22.850022 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:22.850028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:22.850269 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:22.850544 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:22.851850 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:22.853064 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:22.854530 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:22.855099 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:22.855306 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:22.858761 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:22.858769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:22.858781 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:22.858814 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:22.861775 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:22.862550 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:22.862556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:22.862558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:22.862711 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:22.863120 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:22.863129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:22.863132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:22.863635 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:22.863852 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:22.864493 (+ 641us) server_negotiation.cc:299] Negotiation successful
0923 21:06:22.864871 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":1456,"spinlock_wait_cycles":26880,"thread_start_us":1226,"threads_started":1}
I20260923 21:06:22.869519 22895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:22.870692 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.870251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:49998)
0923 21:06:22.870465 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:22.870574 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 21:06:22.870970 22845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
I20260923 21:06:22.872643 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.872281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50004)
0923 21:06:22.872451 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:22.872516 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 21:06:22.872867 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39853: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 21:06:22.898288 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39853
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45209,127.0.97.61:39853,127.0.97.60:35159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:06:22.922257 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } attempt: 1
I20260923 21:06:22.923676 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.923171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50010)
0923 21:06:22.923397 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:22.923493 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 21:06:22.924867 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.924414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50018)
0923 21:06:22.924637 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:22.924706 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 21:06:22.925113 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39853: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
I20260923 21:06:22.993424 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } attempt: 2
I20260923 21:06:22.994774 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.994202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50034)
0923 21:06:22.994517 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:22.994628 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 21:06:22.995900 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:22.995484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50040)
0923 21:06:22.995633 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:22.995743 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 21:06:22.996160 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39853: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
I20260923 21:06:23.087489 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } attempt: 3
I20260923 21:06:23.088825 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.088358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50050)
0923 21:06:23.088577 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.088681 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 21:06:23.089931 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.089544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50058)
0923 21:06:23.089700 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.089797 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 21:06:23.090124 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39853: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
I20260923 21:06:23.233383 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } attempt: 4
I20260923 21:06:23.234735 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.234071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50066)
0923 21:06:23.234414 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.234512 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260923 21:06:23.235735 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.235383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50078)
0923 21:06:23.235539 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.235623 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 21:06:23.235918 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:39853: Network error: Client connection negotiation failed: client connection to 127.0.97.61:39853: connect: Connection refused (error 111)
W20260923 21:06:23.256496 22902 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 21:06:23.256935 22902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:23.257086 22902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:23.257234 22902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:23.265045 22902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:23.265194 22902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:23.265308 22902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:23.265398 22902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:23.265491 22902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:23.276793 22902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45209,127.0.97.61:39853,127.0.97.60:35159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:39853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:23.278278 22902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:23.279697 22902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:23.292810 22907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:23.293066 22910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:23.293133 22908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:23.294477 22902 server_base.cc:1034] running on GCE node
I20260923 21:06:23.295306 22902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:23.296867 22902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:23.298151 22902 hybrid_clock.cc:648] HybridClock initialized: now 1790197583298092 us; error 39 us; skew 500 ppm
I20260923 21:06:23.298590 22902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:23.302263 22902 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 21:06:23.304015 22902 webserver.cc:460] Webserver started at http://127.0.97.61:38531/ using document root <none> and password file <none>
I20260923 21:06:23.304829 22902 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:23.304979 22902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:23.305384 22902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:23.308718 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data/instance:
uuid: "b914c45d5a0f46e88c620ace1047097e"
format_stamp: "Formatted at 2026-09-23 21:06:23 on dist-test-slave-8xn5"
server_key: "74bbc20d504960950939ff609b4dc020"
server_key_iv: "60bd2acd84be262061144184c83ad36a"
server_key_version: "encryptionkey@0"
I20260923 21:06:23.309598 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/wal/instance:
uuid: "b914c45d5a0f46e88c620ace1047097e"
format_stamp: "Formatted at 2026-09-23 21:06:23 on dist-test-slave-8xn5"
server_key: "74bbc20d504960950939ff609b4dc020"
server_key_iv: "60bd2acd84be262061144184c83ad36a"
server_key_version: "encryptionkey@0"
I20260923 21:06:23.315595 22902 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 21:06:23.319510 22917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:23.321128 22902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:06:23.321344 22902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/wal
uuid: "b914c45d5a0f46e88c620ace1047097e"
format_stamp: "Formatted at 2026-09-23 21:06:23 on dist-test-slave-8xn5"
server_key: "74bbc20d504960950939ff609b4dc020"
server_key_iv: "60bd2acd84be262061144184c83ad36a"
server_key_version: "encryptionkey@0"
I20260923 21:06:23.321593 22902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:23.334311 22902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:23.339140 22902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:23.339475 22902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:23.359001 22902 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:39853
I20260923 21:06:23.359023 22968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:39853 every 8 connection(s)
I20260923 21:06:23.360461 22902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
I20260923 21:06:23.364775 22969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:23.365718 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22902
I20260923 21:06:23.365952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-1/wal/instance
I20260923 21:06:23.366576 388 external_mini_cluster.cc:1428] Setting key 5e91e8277a634abf2313d54ab167ea0a
I20260923 21:06:23.376597 22969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 21:06:23.400828 22972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.380368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.0.97.62:45209)
0923 21:06:23.380902 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.380908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.382010 (+ 1102us) server_negotiation.cc:407] Connection header received
0923 21:06:23.384210 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.384216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.384328 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.384459 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.386408 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.387194 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.388861 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.389324 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.389467 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.393610 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.393617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.393620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.393674 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.397049 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.397971 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.397977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.397981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.398126 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.398710 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.398717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.398720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.399043 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.399216 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.400119 (+ 903us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.400399 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":306,"threads_started":1}
I20260923 21:06:23.401942 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.380951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45209 (local address 127.0.0.1:59382)
0923 21:06:23.381834 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.381876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:23.383911 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:23.384727 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:23.384737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:23.385415 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:23.386120 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.386142 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.387368 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.387382 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.388690 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.388702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.389579 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.389588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.389904 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.390846 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:23.390872 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:23.393311 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:23.397251 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.397263 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.397284 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.397819 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.398255 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.398261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.398265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.398579 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.399193 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:23.399203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:23.399663 (+ 460us) client_negotiation.cc:769] Sending connection context
0923 21:06:23.399976 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 21:06:23.400659 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":342,"threads_started":1}
I20260923 21:06:23.404121 22969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:06:23.420521 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.405067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50084)
0923 21:06:23.405300 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.405318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:23.405440 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:23.406534 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:23.406539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:23.407021 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:23.407185 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.407195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.408750 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.408758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.409688 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.409699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.410693 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.410700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.410893 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.411575 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:23.411596 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:23.413946 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:23.417670 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.417678 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.417686 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.418100 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.418555 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.418560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.418564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.418719 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.419401 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:23.419405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:23.419554 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 21:06:23.419709 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 21:06:23.420029 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 21:06:23.420576 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.405272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.0.97.61:39853)
0923 21:06:23.405997 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.406004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.406031 (+ 27us) server_negotiation.cc:407] Connection header received
0923 21:06:23.406083 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.406090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.406322 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.406554 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.407371 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.408575 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.409835 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.410501 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.410673 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.414193 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.414206 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.414222 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.414255 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.417505 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.418252 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.418260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.418262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.418438 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.418840 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.418849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.418852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.419235 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.419460 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.419868 (+ 408us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.420157 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":569,"thread_start_us":427,"threads_started":1}
I20260923 21:06:23.423352 22969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:23.424767 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.424161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41280)
0923 21:06:23.424332 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.424457 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 21:06:23.425055 22919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
I20260923 21:06:23.427007 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.426527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41286)
0923 21:06:23.426810 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.426895 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260923 21:06:23.427251 22969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 21:06:23.431087 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35159
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45209,127.0.97.61:39853,127.0.97.60:35159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:06:23.476641 22969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 1
I20260923 21:06:23.477885 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.477423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41296)
0923 21:06:23.477622 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.477722 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 21:06:23.479441 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.479040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41306)
0923 21:06:23.479226 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.479322 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 21:06:23.479728 22969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
I20260923 21:06:23.535190 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } attempt: 5
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:06:23.547984 22969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 2
I20260923 21:06:23.549077 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.548610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41320)
0923 21:06:23.548795 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.548894 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 21:06:23.550256 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.549827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41334)
0923 21:06:23.550008 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.550088 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 21:06:23.550475 22969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
I20260923 21:06:23.551581 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.536085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50086 (local address 127.0.97.61:39853)
0923 21:06:23.536300 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.536306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.536332 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:06:23.536484 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.536489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.536582 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.536707 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.537528 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.538682 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.539894 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.540604 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.540733 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.544314 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.544320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.544325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.544368 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.548452 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.549296 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.549302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.549305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.549459 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.550053 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.550058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.550061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.550620 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.550778 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.551216 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.551393 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 21:06:23.551894 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.535987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50086)
0923 21:06:23.536190 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.536212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:23.536341 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:23.536759 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:23.536762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:23.537160 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:23.537344 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.537352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.538836 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.538840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.539732 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.539744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.540764 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.540768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.540933 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.541709 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:23.541727 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:23.544058 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:23.548633 (+ 4575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.548638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.548643 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.549099 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.549735 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.549740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.549744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.549922 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.550761 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:23.550766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:23.550951 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 21:06:23.551126 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 21:06:23.551467 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 21:06:23.552960 22895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:23.554037 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.553538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41348)
0923 21:06:23.553750 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.553861 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 21:06:23.555037 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.554674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41362)
0923 21:06:23.554821 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.554919 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 21:06:23.555280 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
I20260923 21:06:23.606528 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 1
I20260923 21:06:23.607977 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.607310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41378)
0923 21:06:23.607592 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.607732 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 21:06:23.609153 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.608787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41388)
0923 21:06:23.608942 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.609030 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 21:06:23.609377 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
I20260923 21:06:23.641726 22969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 3
I20260923 21:06:23.642932 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.642447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41398)
0923 21:06:23.642687 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.642774 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 21:06:23.644049 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.643644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41402)
0923 21:06:23.643869 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.643940 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260923 21:06:23.644345 22969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
I20260923 21:06:23.677618 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 2
I20260923 21:06:23.679005 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.678318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41418)
0923 21:06:23.678658 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.678784 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 21:06:23.680217 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.679769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41432)
0923 21:06:23.679940 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.680044 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 21:06:23.680480 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
W20260923 21:06:23.784209 22977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 21:06:23.784580 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:23.784690 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:23.784791 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 21:06:23.786702 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 3
I20260923 21:06:23.787667 22969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 4
I20260923 21:06:23.787864 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.787331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41440)
0923 21:06:23.787556 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.787668 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 21:06:23.788724 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.788262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41442)
0923 21:06:23.788491 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.788579 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 21:06:23.789474 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.789091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41452)
0923 21:06:23.789301 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.789374 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 21:06:23.789762 22895 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
I20260923 21:06:23.789848 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.789452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41468)
0923 21:06:23.789633 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.789715 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 21:06:23.790051 22969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35159: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35159: connect: Connection refused (error 111)
W20260923 21:06:23.793437 22977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:23.793567 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:23.793687 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:23.793780 22977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:23.793869 22977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:23.804266 22977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45209,127.0.97.61:39853,127.0.97.60:35159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:23.805704 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:23.807158 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:23.819276 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:23.819317 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:23.819681 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:23.820482 22977 server_base.cc:1034] running on GCE node
I20260923 21:06:23.821272 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:23.822779 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:23.824007 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790197583823930 us; error 49 us; skew 500 ppm
I20260923 21:06:23.824375 22977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:23.834832 22977 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 21:06:23.836483 22977 webserver.cc:460] Webserver started at http://127.0.97.60:45339/ using document root <none> and password file <none>
I20260923 21:06:23.837244 22977 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:23.837366 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:23.837710 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:23.840777 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data/instance:
uuid: "016dba0e26d24ebcb9bfef170f0c07f2"
format_stamp: "Formatted at 2026-09-23 21:06:23 on dist-test-slave-8xn5"
server_key: "347db5e08fe062411a155c8fcb271ded"
server_key_iv: "18a96ce8738a96883f0a39ac784f4819"
server_key_version: "encryptionkey@0"
I20260923 21:06:23.841595 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/wal/instance:
uuid: "016dba0e26d24ebcb9bfef170f0c07f2"
format_stamp: "Formatted at 2026-09-23 21:06:23 on dist-test-slave-8xn5"
server_key: "347db5e08fe062411a155c8fcb271ded"
server_key_iv: "18a96ce8738a96883f0a39ac784f4819"
server_key_version: "encryptionkey@0"
I20260923 21:06:23.846976 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 21:06:23.850535 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:23.851892 22977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:06:23.852087 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/wal
uuid: "016dba0e26d24ebcb9bfef170f0c07f2"
format_stamp: "Formatted at 2026-09-23 21:06:23 on dist-test-slave-8xn5"
server_key: "347db5e08fe062411a155c8fcb271ded"
server_key_iv: "18a96ce8738a96883f0a39ac784f4819"
server_key_version: "encryptionkey@0"
I20260923 21:06:23.852306 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:23.912473 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:23.917462 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:23.917801 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:23.939137 22977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35159
I20260923 21:06:23.939160 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35159 every 8 connection(s)
I20260923 21:06:23.940732 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
I20260923 21:06:23.944773 23044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:23.948532 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 22977
I20260923 21:06:23.948755 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790197207951953-388-0/minicluster-data/master-2/wal/instance
I20260923 21:06:23.949271 388 external_mini_cluster.cc:1428] Setting key 1e579fcaa5ca486b303f76a5e10d37c7
I20260923 21:06:23.952375 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:06:23.966969 22895 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 4
I20260923 21:06:23.976290 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.956668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.0.97.62:45209)
0923 21:06:23.957177 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.957185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.957729 (+ 544us) server_negotiation.cc:407] Connection header received
0923 21:06:23.957926 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.957931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.958043 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.958174 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.959525 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.960300 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.962029 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.962721 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.962880 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.967964 (+ 5084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.967971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.967975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.968028 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.971804 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.972598 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.972605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.972608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.972756 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.973304 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.973312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.973318 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.973620 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.973800 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.975615 (+ 1815us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.975858 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260923 21:06:23.977085 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.951473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.0.97.60:35159)
0923 21:06:23.952380 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.952393 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.952442 (+ 49us) server_negotiation.cc:407] Connection header received
0923 21:06:23.954557 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.954584 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.955095 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.955553 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.957563 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.959347 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.961059 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.961881 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.962136 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.967435 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.967445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.967460 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.967498 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.972134 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.973082 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.973091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.973095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.973280 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.973743 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.973748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.973751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.974318 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.974699 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.975337 (+ 638us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.975670 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":669,"thread_start_us":426,"threads_started":1}
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 21:06:23.977941 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.956741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45209 (local address 127.0.0.1:59384)
0923 21:06:23.957568 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.957617 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:23.957773 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:23.958222 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:23.958227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:23.958966 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:23.959327 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.959340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.960472 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.960478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.961861 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.961874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.962981 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.962988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.963247 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.963865 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:23.963898 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:23.967711 (+ 3813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:23.971948 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.971961 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.971976 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.972467 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.972911 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.972918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.972923 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.973170 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.974442 (+ 1272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:23.974450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:23.974768 (+ 318us) client_negotiation.cc:769] Sending connection context
0923 21:06:23.975085 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 21:06:23.977494 (+ 2409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":261,"thread_start_us":427,"threads_started":1}
I20260923 21:06:23.979882 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:23.987140 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.967694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41480)
0923 21:06:23.968020 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.968039 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:23.968146 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:23.970103 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:23.970110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:23.970753 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:23.971055 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.971070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.972855 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.972861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.974203 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.974215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.975078 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.975083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.975279 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.975914 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:23.975933 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:23.978662 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:23.982933 (+ 4271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.982939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.982945 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.983546 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.985090 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.985096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.985099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.985257 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.985902 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:23.985955 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:23.986162 (+ 207us) client_negotiation.cc:769] Sending connection context
0923 21:06:23.986309 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 21:06:23.986695 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260923 21:06:23.987234 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.967987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.0.97.60:35159)
0923 21:06:23.969701 (+ 1714us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.969708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.969743 (+ 35us) server_negotiation.cc:407] Connection header received
0923 21:06:23.969805 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.969810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.969900 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.970651 (+ 751us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.971291 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.972701 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.974427 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.974918 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.975057 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.978870 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.978877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.978882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.978920 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.982774 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.984717 (+ 1943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.984721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.984724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.984966 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.985380 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.985387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.985394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.985772 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.985978 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.986517 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.986783 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562,"thread_start_us":440,"threads_started":1}
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 23 21:06:23 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:06:23.994727 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3: Bootstrap starting.
I20260923 21:06:23.997587 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.980645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:39853 (local address 127.0.0.1:50098)
0923 21:06:23.981118 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:23.981141 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:23.981249 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:23.982317 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:23.982322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:23.982874 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:23.983037 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.983045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.984246 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.984250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.985981 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:23.985998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.987125 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.987131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:23.987428 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.988277 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:23.988293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:23.990491 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:23.994596 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.994601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.994605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.995161 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.995662 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:23.995667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:23.995670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:23.995840 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:23.996499 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:23.996504 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:23.996733 (+ 229us) client_negotiation.cc:769] Sending connection context
0923 21:06:23.996885 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 21:06:23.997197 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260923 21:06:23.997617 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.981076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.0.97.61:39853)
0923 21:06:23.981264 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.981270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.981290 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:06:23.981355 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.981360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.981440 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.981561 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.983196 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.984070 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.986146 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.986794 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.986922 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.990692 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.990697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.990700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.990735 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.994414 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.995314 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.995321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.995325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.995496 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.995972 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.995980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.995983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.996382 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.996538 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.996973 (+ 435us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.997257 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 21:06:23.999064 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.981674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.0.97.62:45209)
0923 21:06:23.981853 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:23.981858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:23.981876 (+ 18us) server_negotiation.cc:407] Connection header received
0923 21:06:23.981928 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:23.981933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:23.982018 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:23.982133 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:23.983699 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.984576 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.986475 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:23.987190 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:23.987323 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:23.992418 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:23.992423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:23.992426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:23.992457 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:23.995498 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.996335 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.996343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.996348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.996497 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:23.996995 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:23.997002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:23.997005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:23.997403 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:23.997683 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:23.998566 (+ 883us) server_negotiation.cc:299] Negotiation successful
0923 21:06:23.998814 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:06:23.999117 23044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:23.999595 22895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:24.001088 22895 log.cc:824] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3: Log is configured to *not* fsync() on all Append() calls
Sep 23 21:06:24 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 21:06:24 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:06:24.011373 22895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3: No bootstrap required, opened a new log
I20260923 21:06:24.015694 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:23.999834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41496)
0923 21:06:24.000006 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:24.000023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:24.000115 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:24.000606 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:24.000610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:24.001093 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:24.001295 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:24.001307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.002413 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.002419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:24.003270 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:24.003280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.004120 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.004130 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:24.004422 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.005287 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:24.005328 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:24.008905 (+ 3577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:24.012938 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:24.012943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:24.012946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:24.013482 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:24.013863 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:24.013869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:24.013873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:24.014000 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:24.014620 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:24.014632 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:24.014792 (+ 160us) client_negotiation.cc:769] Sending connection context
0923 21:06:24.014953 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 21:06:24.015281 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 21:06:24.016374 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.000147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.97.60:35159)
0923 21:06:24.000335 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.000342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.000356 (+ 14us) server_negotiation.cc:407] Connection header received
0923 21:06:24.000399 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.000404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.000489 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.000592 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.001447 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.002266 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.003414 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.003928 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.004055 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.009161 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:24.009167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:24.009171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:24.009208 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:24.012750 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.013614 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.013622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.013626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.013752 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.014103 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.014111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.014114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.014499 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:24.014650 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.015088 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.016159 (+ 1071us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":864,"server-negotiator.queue_time_us":64}
I20260923 21:06:24.019830 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.001217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.0.97.61:39853)
0923 21:06:24.001400 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.001407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.001427 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:06:24.001492 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.001498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.001568 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.001665 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.003791 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.004633 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.005847 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.006276 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.006471 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.011318 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:24.011325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:24.011329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:24.011376 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:24.015149 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.016210 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.016217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.016220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.016385 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.017307 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.017315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.017319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.017680 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:24.017968 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.019225 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.019472 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 21:06:24.021160 23044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: Bootstrap starting.
I20260923 21:06:24.022936 22895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
I20260923 21:06:24.023388 22895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 396bd29b7ed74232a0798ea809ae41e3, State: Initialized, Role: FOLLOWER
I20260923 21:06:24.023933 22895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
I20260923 21:06:24.024669 23044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:24.025004 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.025415 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:24.025645 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:24.026077 23044 log.cc:824] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:24.026535 22895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:24.029176 22870 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "396bd29b7ed74232a0798ea809ae41e3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59386
I20260923 21:06:24.029619 23044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: No bootstrap required, opened a new log
I20260923 21:06:24.029608 22870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 21:06:24.029804 22870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:24.033496 22870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
I20260923 21:06:24.037055 22870 leader_election.cc:288] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [CANDIDATE]: Term 1 election: Requested vote from peers b914c45d5a0f46e88c620ace1047097e (127.0.97.61:39853), 016dba0e26d24ebcb9bfef170f0c07f2 (127.0.97.60:35159)
I20260923 21:06:24.039436 23044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
I20260923 21:06:24.039981 23044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 016dba0e26d24ebcb9bfef170f0c07f2, State: Initialized, Role: FOLLOWER
I20260923 21:06:24.040545 23044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
W20260923 21:06:24.041499 23064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 21:06:24.041702 23064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 21:06:24.042397 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.042804 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:24.043883 23044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:24.046447 23019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "396bd29b7ed74232a0798ea809ae41e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "016dba0e26d24ebcb9bfef170f0c07f2"
I20260923 21:06:24.046975 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:24.050877 23019 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 396bd29b7ed74232a0798ea809ae41e3 in term 1.
I20260923 21:06:24.052073 22845 leader_election.cc:302] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 016dba0e26d24ebcb9bfef170f0c07f2, 396bd29b7ed74232a0798ea809ae41e3; no voters:
I20260923 21:06:24.052635 23052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:24.057052 23052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [term 1 LEADER]: Becoming Leader. State: Replica: 396bd29b7ed74232a0798ea809ae41e3, State: Running, Role: LEADER
I20260923 21:06:24.057723 23052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
W20260923 21:06:24.061136 22849 tablet.cc:2369] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:06:24.039757 22828 master_runner.cc:186] Error getting master registration for 127.0.97.61:39853: OK, instance_id { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" instance_seqno: 1790197583353642 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 21:06:24.064896 22828 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 21:06:24.068981 23069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396bd29b7ed74232a0798ea809ae41e3. Latest consensus state: current_term: 1 leader_uuid: "396bd29b7ed74232a0798ea809ae41e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.069388 23069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:24.075392 23075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:24.079535 23075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:24.089241 22969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } attempt: 5
Sep 23 21:06:24 dist-test-slave-8xn5 krb5kdc[22815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 21:06:24.092636 22977 master_runner.cc:186] Error getting master registration for 127.0.97.61:39853: OK, instance_id { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" instance_seqno: 1790197583353642 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 21:06:24.106083 22977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 21:06:24.108232 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.089890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35159 (local address 127.0.0.1:41502)
0923 21:06:24.090247 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:24.090268 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:24.090410 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:24.091299 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:24.091303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:24.091868 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:24.092086 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:24.092098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.093620 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.093635 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:24.095081 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:24.095097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.096407 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.096415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:24.096654 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.097358 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:24.097383 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:24.100299 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:24.104545 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:24.104553 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:24.104559 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:24.105181 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:24.105685 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:24.105690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:24.105693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:24.105875 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:24.106807 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:24.106815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:24.107041 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 21:06:24.107219 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 21:06:24.107658 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260923 21:06:24.109524 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.090642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41502 (local address 127.0.97.60:35159)
0923 21:06:24.090814 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.090819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.090836 (+ 17us) server_negotiation.cc:407] Connection header received
0923 21:06:24.090885 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.090890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.090989 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.091080 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.092260 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.093437 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.095260 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.096014 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.096154 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.100526 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:24.100534 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:24.100540 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:24.100585 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:24.104313 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.105334 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.105342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.105347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.105535 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.106006 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.106011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.106014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.106506 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:24.106727 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.108749 (+ 2022us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.109068 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:06:24.115692 22969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e: Bootstrap starting.
W20260923 21:06:24.116853 23084 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 21:06:24.116981 23084 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 21:06:24.119844 22969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:24.121263 22969 log.cc:824] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:24.126956 22969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e: No bootstrap required, opened a new log
I20260923 21:06:24.147230 22969 raft_consensus.cc:348] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
I20260923 21:06:24.147769 22969 raft_consensus.cc:729] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b914c45d5a0f46e88c620ace1047097e, State: Initialized, Role: FOLLOWER
I20260923 21:06:24.148378 22969 consensus_queue.cc:260] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } }
I20260923 21:06:24.149217 23019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [term 1 FOLLOWER]: Refusing update from remote peer 396bd29b7ed74232a0798ea809ae41e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:06:24.150310 23069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:06:24.149897 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.150707 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:24.150936 22969 sys_catalog.cc:564] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:24.151319 22943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:24.155153 22943 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [term 1 FOLLOWER]: Refusing update from remote peer 396bd29b7ed74232a0798ea809ae41e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:06:24.155337 22944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "396bd29b7ed74232a0798ea809ae41e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b914c45d5a0f46e88c620ace1047097e"
I20260923 21:06:24.159459 22944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 396bd29b7ed74232a0798ea809ae41e3 in term 1.
I20260923 21:06:24.160668 23069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 21:06:24.178431 22998 tablet.cc:2369] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:06:24.188012 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396bd29b7ed74232a0798ea809ae41e3. Latest consensus state: current_term: 1 leader_uuid: "396bd29b7ed74232a0798ea809ae41e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.188385 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:24.197309 23069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "396bd29b7ed74232a0798ea809ae41e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.197594 23069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:24.201538 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 396bd29b7ed74232a0798ea809ae41e3. Latest consensus state: current_term: 1 leader_uuid: "396bd29b7ed74232a0798ea809ae41e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.201905 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:24.202833 23075 catalog_manager.cc:1347] Generated new cluster ID: a5c3ce1747e84bd9a26f215339df3f6c
I20260923 21:06:24.202970 23075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:24.205372 23069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "396bd29b7ed74232a0798ea809ae41e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.205669 23069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:24.207990 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "396bd29b7ed74232a0798ea809ae41e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.208256 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:24.215754 23090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "396bd29b7ed74232a0798ea809ae41e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "396bd29b7ed74232a0798ea809ae41e3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45209 } } peers { permanent_uuid: "b914c45d5a0f46e88c620ace1047097e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 39853 } } peers { permanent_uuid: "016dba0e26d24ebcb9bfef170f0c07f2" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35159 } } }
I20260923 21:06:24.216037 23090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:24.217371 23075 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:24.219241 23075 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:24.227947 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.125145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.0.97.61:39853)
0923 21:06:24.125497 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.125503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.162389 (+ 36886us) server_negotiation.cc:407] Connection header received
0923 21:06:24.163029 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.163034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.163104 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.163196 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.173375 (+ 10179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.174941 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.192504 (+ 17563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.193419 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.193550 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.209567 (+ 16017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:24.209580 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:24.209588 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:24.209649 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:24.213471 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.222960 (+ 9489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.222970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.222976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.223171 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.223712 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.223720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.223726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.224203 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:24.224521 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.227037 (+ 2516us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.227411 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260923 21:06:24.236033 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.124949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.97.62:45209)
0923 21:06:24.125339 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.125347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.156867 (+ 31520us) server_negotiation.cc:407] Connection header received
0923 21:06:24.158633 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.158642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.158752 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.158845 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.166419 (+ 7574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.167959 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.175393 (+ 7434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.176361 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.176498 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.201418 (+ 24920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:24.201425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:24.201430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:24.201504 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:24.206207 (+ 4703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.219844 (+ 13637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.219857 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.219864 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.220240 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.225146 (+ 4906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.225155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.225161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.225779 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:24.226137 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.235288 (+ 9151us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.235591 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260923 21:06:24.239890 23075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 396bd29b7ed74232a0798ea809ae41e3: Generated new TSK 0
I20260923 21:06:24.241395 23075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:24.244877 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.157660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.0.97.60:35159)
0923 21:06:24.157814 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.157819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.166256 (+ 8437us) server_negotiation.cc:407] Connection header received
0923 21:06:24.170721 (+ 4465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.170732 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.170845 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.170933 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.178541 (+ 7608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.179689 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.197033 (+ 17344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.197745 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.197853 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.230816 (+ 32963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:24.230823 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:24.230827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:24.230876 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:24.234975 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.239501 (+ 4526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.239509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.239515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.239738 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:24.241907 (+ 2169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:24.241916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:24.241922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:24.242520 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:24.242912 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.243776 (+ 864us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.244172 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 21:06:24.341840 23113 catalog_manager.cc:1259] Loaded cluster ID: a5c3ce1747e84bd9a26f215339df3f6c
I20260923 21:06:24.341990 23113 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e: loading cluster ID for follower catalog manager: success
I20260923 21:06:24.350088 23113 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e: acquiring CA information for follower catalog manager: success
I20260923 21:06:24.353199 23113 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b914c45d5a0f46e88c620ace1047097e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 21:06:24.388336 23048 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.352026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59406 (local address 127.0.97.62:45209)
0923 21:06:24.352219 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.352226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.383284 (+ 31058us) server_negotiation.cc:407] Connection header received
0923 21:06:24.386599 (+ 3315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.386607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.386702 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.386777 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.387994 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59406: BlockingRecv error: recv got EOF from 127.0.0.1:59406 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 21:06:24.390846 22975 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.354307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.0.97.61:39853)
0923 21:06:24.354506 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.354512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.389039 (+ 34527us) server_negotiation.cc:407] Connection header received
0923 21:06:24.389326 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.389333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.389419 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.389490 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.390610 (+ 1120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50116: BlockingRecv error: recv got EOF from 127.0.0.1:50116 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 21:06:24.393982 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.383963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.0.97.60:35159)
0923 21:06:24.384187 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.384193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.391659 (+ 7466us) server_negotiation.cc:407] Connection header received
0923 21:06:24.392554 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.392561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.392650 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.392725 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.393700 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41528: BlockingRecv error: recv got EOF from 127.0.0.1:41528 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 21:06:24.449005 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.428886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.0.97.61:39853)
0923 21:06:24.429085 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.429091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.439631 (+ 10540us) server_negotiation.cc:407] Connection header received
0923 21:06:24.440178 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.440199 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.440457 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.440616 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:24.441349 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.442555 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.444848 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.445592 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.445714 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.447426 (+ 1712us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:24.448168 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:24.448296 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.448548 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.448657 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 21:06:24.497397 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.484910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.0.97.60:35159)
0923 21:06:24.485137 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.485144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.495140 (+ 9996us) server_negotiation.cc:407] Connection header received
0923 21:06:24.495709 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.495716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.495889 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.495973 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.497133 (+ 1160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41540: BlockingRecv error: recv got EOF from 127.0.0.1:41540 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 21:06:24.563568 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.545725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.0.97.61:39853)
0923 21:06:24.545990 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.546004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.556470 (+ 10466us) server_negotiation.cc:407] Connection header received
0923 21:06:24.556785 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.556792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.556914 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.557004 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:24.557790 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.559051 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.561177 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.561846 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.561950 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.562616 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:24.562809 (+ 193us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:24.562873 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.563128 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.563221 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 21:06:24.617306 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.603962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.0.97.60:35159)
0923 21:06:24.604185 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.604192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.615504 (+ 11312us) server_negotiation.cc:407] Connection header received
0923 21:06:24.615831 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.615838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.615930 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.616002 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.617021 (+ 1019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41554: BlockingRecv error: recv got EOF from 127.0.0.1:41554 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:06:24.681450 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.661205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.0.97.61:39853)
0923 21:06:24.661474 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.661485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.673881 (+ 12396us) server_negotiation.cc:407] Connection header received
0923 21:06:24.674274 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.674282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.674438 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.674527 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:24.675347 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.676602 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.678809 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.679576 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.679703 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.680398 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:24.680555 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:24.680630 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.680900 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.681040 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 21:06:24.732846 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.719621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.0.97.60:35159)
0923 21:06:24.719833 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.719840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.730682 (+ 10842us) server_negotiation.cc:407] Connection header received
0923 21:06:24.731281 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.731290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.731406 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.731495 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.732568 (+ 1073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41558: BlockingRecv error: recv got EOF from 127.0.0.1:41558 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 21:06:24.790518 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.777041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.0.97.61:39853)
0923 21:06:24.777279 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.777289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.783895 (+ 6606us) server_negotiation.cc:407] Connection header received
0923 21:06:24.784145 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.784151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.784258 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.784379 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:24.784930 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.786180 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.788186 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.788880 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.789032 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.789590 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:24.789742 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:24.789847 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.790057 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.790151 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 21:06:24.800094 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.797708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.0.97.60:35159)
0923 21:06:24.797943 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.797952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.798378 (+ 426us) server_negotiation.cc:407] Connection header received
0923 21:06:24.798575 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.798581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.798714 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.798848 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.799810 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41568: BlockingRecv error: recv got EOF from 127.0.0.1:41568 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 21:06:24.820153 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.812726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.0.97.61:39853)
0923 21:06:24.812949 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.812955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.813593 (+ 638us) server_negotiation.cc:407] Connection header received
0923 21:06:24.813820 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.813826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.813932 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.814047 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:24.814564 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.815788 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.817732 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.818505 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.818652 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.819158 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:24.819347 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:24.819442 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.819656 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.819762 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 21:06:24.857048 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.837611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.0.97.60:35159)
0923 21:06:24.837818 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.837825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.855031 (+ 17206us) server_negotiation.cc:407] Connection header received
0923 21:06:24.855234 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.855241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.855393 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.855547 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.856778 (+ 1231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41572: BlockingRecv error: recv got EOF from 127.0.0.1:41572 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 21:06:24.886622 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.878269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.0.97.61:39853)
0923 21:06:24.878494 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.878501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.879100 (+ 599us) server_negotiation.cc:407] Connection header received
0923 21:06:24.879328 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.879335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.879437 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.879566 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:24.880123 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.881659 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.883992 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.884768 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.884927 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.885625 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:24.885799 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:24.885908 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.886124 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.886232 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260923 21:06:24.896441 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.894495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.0.97.60:35159)
0923 21:06:24.894709 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.894715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.894740 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:06:24.895046 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.895053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.895154 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.895270 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.896164 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41576: BlockingRecv error: recv got EOF from 127.0.0.1:41576 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:06:24.938304 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:24.932226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.97.61:39853)
0923 21:06:24.932639 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.932648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.932876 (+ 228us) server_negotiation.cc:407] Connection header received
0923 21:06:24.933191 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.933196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.933270 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.933387 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:24.933882 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.934779 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.936272 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:24.936858 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:24.937015 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:24.937464 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:24.937594 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:24.937709 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:24.937877 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 21:06:24.937967 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260923 21:06:24.947186 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:24.944862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.0.97.60:35159)
0923 21:06:24.945125 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:24.945132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:24.945471 (+ 339us) server_negotiation.cc:407] Connection header received
0923 21:06:24.945653 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:24.945658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:24.945735 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:24.945828 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:24.946942 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41580: BlockingRecv error: recv got EOF from 127.0.0.1:41580 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 21:06:25.020684 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:25.014846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.0.97.61:39853)
0923 21:06:25.015112 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:25.015119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:25.015563 (+ 444us) server_negotiation.cc:407] Connection header received
0923 21:06:25.015693 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:25.015703 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:25.015827 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:25.015984 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:25.016524 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.017398 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.018901 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.019329 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.019432 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:25.019888 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:25.020007 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:25.020078 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:25.020338 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 21:06:25.020407 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260923 21:06:25.028991 23050 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:25.026928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.97.60:35159)
0923 21:06:25.027215 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:25.027223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:25.027484 (+ 261us) server_negotiation.cc:407] Connection header received
0923 21:06:25.027707 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:25.027715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:25.027793 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:25.027903 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:25.028744 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41588: BlockingRecv error: recv got EOF from 127.0.0.1:41588 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260923 21:06:25.119030 23084 catalog_manager.cc:1259] Loaded cluster ID: a5c3ce1747e84bd9a26f215339df3f6c
I20260923 21:06:25.119145 23084 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: loading cluster ID for follower catalog manager: success
I20260923 21:06:25.121568 23084 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: acquiring CA information for follower catalog manager: success
I20260923 21:06:25.123469 23084 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 016dba0e26d24ebcb9bfef170f0c07f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 21:06:25.166627 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:25.161287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.0.97.61:39853)
0923 21:06:25.161539 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:25.161548 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:25.161877 (+ 329us) server_negotiation.cc:407] Connection header received
0923 21:06:25.162146 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:25.162150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:25.162221 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:25.162331 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:25.162780 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.163489 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.164922 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.165363 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.165499 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:25.165892 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:25.166012 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:25.166120 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:25.166260 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 21:06:25.166363 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 21:06:25.179690 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:25.172972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.97.60:35159)
0923 21:06:25.173265 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:25.173273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:25.173493 (+ 220us) server_negotiation.cc:407] Connection header received
0923 21:06:25.173826 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:25.173831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:25.174022 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:25.174127 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:25.174675 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.175542 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.177053 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.177487 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.177599 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:25.178159 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:25.178890 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:25.179030 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:25.179197 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 21:06:25.179270 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 21:06:25.195086 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:25.187070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.97.62:45209)
0923 21:06:25.187965 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:25.187974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:25.188000 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:06:25.188169 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:25.188173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:25.188364 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:25.188566 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:25.189108 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.189888 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.192130 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.192781 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.192936 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:25.193585 (+ 649us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:25.194125 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:25.194261 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:25.194464 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 21:06:25.194562 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":639,"threads_started":1}
I20260923 21:06:25.198365 22975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:25.187700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.97.61:39853)
0923 21:06:25.187898 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:25.187902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:25.191077 (+ 3175us) server_negotiation.cc:407] Connection header received
0923 21:06:25.191299 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:25.191304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:25.191377 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:25.191526 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:25.192291 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.193478 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.195548 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.196107 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.196239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:25.197457 (+ 1218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:25.197641 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:25.197745 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:25.197938 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 21:06:25.198033 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 21:06:25.199610 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:25.191338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.97.60:35159)
0923 21:06:25.191614 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:25.191621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:25.192584 (+ 963us) server_negotiation.cc:407] Connection header received
0923 21:06:25.192764 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:25.192770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:25.192881 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:25.192968 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:25.193731 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.194910 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.197165 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:25.197849 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:25.197997 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:25.198727 (+ 730us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:25.198874 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:25.199018 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:25.199203 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 21:06:25.199289 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 21:06:25.200402 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22828
I20260923 21:06:25.201820 22890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:25.317157 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22828
I20260923 21:06:25.339104 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22902
I20260923 21:06:25.340199 22964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:25.450083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22902
I20260923 21:06:25.470394 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 22977
I20260923 21:06:25.471690 23039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:25.584223 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 22977
2026-09-23T21:06:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4693 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 21:06:25 dist-test-slave-8xn5 krb5kdc[23225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 21:06:25 dist-test-slave-8xn5 krb5kdc[23225](info): set up 2 sockets
Sep 23 21:06:25 dist-test-slave-8xn5 krb5kdc[23225](info): commencing operation
krb5kdc: starting...
W20260923 21:06:26.791286 388 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/test-user.keytab.
Sep 23 21:06:26 dist-test-slave-8xn5 krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T21:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T21:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260923 21:06:27.083343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38395
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38395,127.0.97.61:40277,127.0.97.60:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 21:06:27.435884 23238 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 21:06:27.436290 23238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:27.436437 23238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:27.436590 23238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:27.444898 23238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:27.445014 23238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:27.445102 23238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:27.445195 23238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:27.445300 23238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:27.455703 23238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38395,127.0.97.61:40277,127.0.97.60:33885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:27.457113 23238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:27.458555 23238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:27.471240 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:27.471289 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:27.471197 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:27.472198 23238 server_base.cc:1034] running on GCE node
I20260923 21:06:27.472995 23238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:27.474498 23238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:27.475749 23238 hybrid_clock.cc:648] HybridClock initialized: now 1790197587475689 us; error 29 us; skew 500 ppm
I20260923 21:06:27.476148 23238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:27 dist-test-slave-8xn5 krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:27.479732 23238 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260923 21:06:27.481366 23238 webserver.cc:460] Webserver started at http://127.0.97.62:41807/ using document root <none> and password file <none>
I20260923 21:06:27.482064 23238 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:27.482151 23238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:27.482492 23238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:27.485491 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data/instance:
uuid: "f8e1ca3559724c77be157bf0a8da6856"
format_stamp: "Formatted at 2026-09-23 21:06:27 on dist-test-slave-8xn5"
server_key: "e924231eb5ce9cc8d0cbd611d73139ea"
server_key_iv: "d7145a1abcc89ca77465c064e0e533ce"
server_key_version: "encryptionkey@0"
I20260923 21:06:27.486251 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/wal/instance:
uuid: "f8e1ca3559724c77be157bf0a8da6856"
format_stamp: "Formatted at 2026-09-23 21:06:27 on dist-test-slave-8xn5"
server_key: "e924231eb5ce9cc8d0cbd611d73139ea"
server_key_iv: "d7145a1abcc89ca77465c064e0e533ce"
server_key_version: "encryptionkey@0"
I20260923 21:06:27.491845 23238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:06:27.495527 23253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:27.497030 23238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:06:27.497226 23238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/wal
uuid: "f8e1ca3559724c77be157bf0a8da6856"
format_stamp: "Formatted at 2026-09-23 21:06:27 on dist-test-slave-8xn5"
server_key: "e924231eb5ce9cc8d0cbd611d73139ea"
server_key_iv: "d7145a1abcc89ca77465c064e0e533ce"
server_key_version: "encryptionkey@0"
I20260923 21:06:27.497462 23238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:27.509829 23238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:27.514729 23238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:27.515023 23238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:27.535135 23238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38395
I20260923 21:06:27.535153 23304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38395 every 8 connection(s)
I20260923 21:06:27.536614 23238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/data/info.pb
I20260923 21:06:27.540617 23305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:27.541709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 23238
I20260923 21:06:27.541909 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-0/wal/instance
I20260923 21:06:27.542524 388 external_mini_cluster.cc:1428] Setting key c30e09349fe4b6e2fae1fc3bfd1b13c0
I20260923 21:06:27.551301 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:27 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260923 21:06:27.575461 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.555351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38395 (local address 127.0.0.1:59320)
0923 21:06:27.556131 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.556178 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:27.558474 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:27.559351 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:27.559357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:27.560074 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:27.560410 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:27.560428 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:27.561966 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:27.561971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:27.563098 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:27.563110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:27.564036 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:27.564042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:27.564230 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:27.564946 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:27.564968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:27.567710 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:27.571483 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:27.571493 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:27.571505 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:27.571882 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:27.572301 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:27.572326 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:27.572329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:27.572487 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:27.573242 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:27.573249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:27.573457 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 21:06:27.573703 (+ 246us) client_negotiation.cc:240] Negotiation successful
0923 21:06:27.574414 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":184,"thread_start_us":313,"threads_started":1}
I20260923 21:06:27.575490 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.556459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.0.97.62:38395)
0923 21:06:27.557253 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:27.557261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:27.557314 (+ 53us) server_negotiation.cc:407] Connection header received
0923 21:06:27.558864 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:27.558870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:27.559127 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:27.559343 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:27.560618 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:27.561790 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:27.563233 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:27.563788 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:27.563993 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:27.567970 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:27.567980 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:27.567993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:27.568029 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:27.571317 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:27.572040 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:27.572047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:27.572049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:27.572189 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:27.572609 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:27.572620 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:27.572623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:27.573102 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:27.573306 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:27.573821 (+ 515us) server_negotiation.cc:299] Negotiation successful
0923 21:06:27.574197 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":328,"threads_started":1}
I20260923 21:06:27.578964 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:27.580255 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.579776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55586)
0923 21:06:27.579956 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.580094 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 21:06:27.580559 23254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
I20260923 21:06:27.582458 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.581887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55588)
0923 21:06:27.582104 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.582226 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260923 21:06:27.582679 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40277: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260923 21:06:27.609973 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40277
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38395,127.0.97.61:40277,127.0.97.60:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:06:27.632066 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } attempt: 1
I20260923 21:06:27.633481 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.632873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55602)
0923 21:06:27.633137 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.633290 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 21:06:27.634778 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.634210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55610)
0923 21:06:27.634432 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.634542 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 21:06:27.635046 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40277: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
I20260923 21:06:27.703612 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } attempt: 2
I20260923 21:06:27.704970 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.704387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55620)
0923 21:06:27.704645 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.704773 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 21:06:27.706207 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.705720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55636)
0923 21:06:27.705893 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.706020 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 21:06:27.706471 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40277: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
I20260923 21:06:27.797796 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } attempt: 3
I20260923 21:06:27.799173 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.798622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55648)
0923 21:06:27.798858 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.798984 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 21:06:27.800498 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.799996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55652)
0923 21:06:27.800166 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.800272 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 21:06:27.800745 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40277: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
I20260923 21:06:27.944056 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } attempt: 4
I20260923 21:06:27.945433 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.944759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55658)
0923 21:06:27.945121 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.945245 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260923 21:06:27.946671 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:27.946178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55660)
0923 21:06:27.946383 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:27.946506 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 21:06:27.946904 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40277: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40277: connect: Connection refused (error 111)
W20260923 21:06:27.960664 23312 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 21:06:27.961027 23312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:27.961112 23312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:27.961202 23312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:27.969148 23312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:27.969280 23312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:27.969345 23312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:27.969414 23312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:27.969487 23312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:27.980392 23312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38395,127.0.97.61:40277,127.0.97.60:33885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:27.981851 23312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:27.983434 23312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:27.997264 23318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:27.997325 23320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:27.997949 23317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:27.998281 23312 server_base.cc:1034] running on GCE node
I20260923 21:06:27.999210 23312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:28.000779 23312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:28.002020 23312 hybrid_clock.cc:648] HybridClock initialized: now 1790197588001947 us; error 36 us; skew 500 ppm
I20260923 21:06:28.002573 23312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:28.006263 23312 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260923 21:06:28.008181 23312 webserver.cc:460] Webserver started at http://127.0.97.61:36425/ using document root <none> and password file <none>
I20260923 21:06:28.008895 23312 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:28.008987 23312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:28.009265 23312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:28.012478 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data/instance:
uuid: "577566b8817d48d692229f9970bfd1e6"
format_stamp: "Formatted at 2026-09-23 21:06:28 on dist-test-slave-8xn5"
server_key: "deb451f3271aa6a10b74dc4f8c4241e2"
server_key_iv: "c4f283321d1b93f10955c9281a8083cb"
server_key_version: "encryptionkey@0"
I20260923 21:06:28.013273 23312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/wal/instance:
uuid: "577566b8817d48d692229f9970bfd1e6"
format_stamp: "Formatted at 2026-09-23 21:06:28 on dist-test-slave-8xn5"
server_key: "deb451f3271aa6a10b74dc4f8c4241e2"
server_key_iv: "c4f283321d1b93f10955c9281a8083cb"
server_key_version: "encryptionkey@0"
I20260923 21:06:28.019024 23312 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 21:06:28.022919 23327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:28.024394 23312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 21:06:28.024544 23312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/wal
uuid: "577566b8817d48d692229f9970bfd1e6"
format_stamp: "Formatted at 2026-09-23 21:06:28 on dist-test-slave-8xn5"
server_key: "deb451f3271aa6a10b74dc4f8c4241e2"
server_key_iv: "c4f283321d1b93f10955c9281a8083cb"
server_key_version: "encryptionkey@0"
I20260923 21:06:28.024762 23312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:28.047300 23312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:28.052016 23312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:28.052244 23312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:28.071647 23312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40277
I20260923 21:06:28.071668 23378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40277 every 8 connection(s)
I20260923 21:06:28.073103 23312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/data/info.pb
I20260923 21:06:28.076639 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 23312
I20260923 21:06:28.076833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-1/wal/instance
I20260923 21:06:28.077270 388 external_mini_cluster.cc:1428] Setting key f49e7bd90d308c8b215ef665a6686bc8
I20260923 21:06:28.077461 23379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:28.084867 23379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260923 21:06:28.110638 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.088238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.0.97.62:38395)
0923 21:06:28.088834 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.088844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.089943 (+ 1099us) server_negotiation.cc:407] Connection header received
0923 21:06:28.092024 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.092033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.092142 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.092291 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.094106 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.095538 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.097312 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.098174 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.098382 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.102567 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.102575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.102580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.102648 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.106530 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.107698 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.107703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.107740 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.107917 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.108550 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.108555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.108557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.108889 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.109086 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.110031 (+ 945us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.110226 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":286,"threads_started":1}
I20260923 21:06:28.112146 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.088811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38395 (local address 127.0.0.1:59322)
0923 21:06:28.089743 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.089797 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.091711 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.092562 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.092572 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.093162 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.093827 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.093847 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.095738 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.095742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.097113 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.097134 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.098467 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.098472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.098693 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.099421 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.099444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.102237 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.106742 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.106756 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.106781 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.107531 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.108072 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.108080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.108087 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.108366 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.109055 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.109067 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.109598 (+ 531us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.109924 (+ 326us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.110565 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":387,"threads_started":1}
I20260923 21:06:28.114208 23379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:06:28.129524 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.115226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55670)
0923 21:06:28.115397 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.115418 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.115527 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.116559 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.116563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.117012 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.117136 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.117145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.118620 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.118625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.119453 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.119463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.120288 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.120293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.120470 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.121079 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.121096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.123196 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.126458 (+ 3262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.126465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.126472 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.127041 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.127420 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.127433 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.127440 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.127610 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.128253 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.128265 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.128468 (+ 203us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.128649 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.128963 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 21:06:28.129683 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.115528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.97.61:40277)
0923 21:06:28.116138 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.116144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.116170 (+ 26us) server_negotiation.cc:407] Connection header received
0923 21:06:28.116213 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.116218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.116415 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.116571 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.117319 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.118475 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.119585 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.120120 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.120247 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.123406 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.123416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.123431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.123457 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.126261 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.127177 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.127184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.127186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.127301 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.127730 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.127733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.127735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.128109 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.128334 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.128865 (+ 531us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.129137 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":333,"threads_started":1}
I20260923 21:06:28.132664 23379 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:28.133790 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.133305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57370)
0923 21:06:28.133496 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.133636 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 21:06:28.134069 23328 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.135805 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.135348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57372)
0923 21:06:28.135528 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.135639 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 21:06:28.136130 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260923 21:06:28.142820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
/tmp/dist-test-taskDsUdkt/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33885
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38395,127.0.97.61:40277,127.0.97.60:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 21:06:28.185489 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 1
I20260923 21:06:28.186947 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.186295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57378)
0923 21:06:28.186606 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.186734 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260923 21:06:28.188248 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.187787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57380)
0923 21:06:28.187983 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.188086 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 21:06:28.188634 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.246213 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } attempt: 5
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:06:28.256946 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 2
I20260923 21:06:28.258101 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.257565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57384)
0923 21:06:28.257765 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.257874 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 21:06:28.259378 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.258975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57398)
0923 21:06:28.259142 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.259233 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 21:06:28.259662 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.263149 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.247144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55682)
0923 21:06:28.247485 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.247507 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.247626 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.248050 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.248053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.248523 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.248711 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.248720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.250151 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.250156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.251300 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.251314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.252332 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.252337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.252621 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.253333 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.253353 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.255801 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.260304 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.260309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.260311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.260822 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.261288 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.261293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.261295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.261471 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.262122 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.262126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.262286 (+ 160us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.262606 (+ 320us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.262893 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 21:06:28.263255 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.247323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.0.97.61:40277)
0923 21:06:28.247542 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.247548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.247573 (+ 25us) server_negotiation.cc:407] Connection header received
0923 21:06:28.247765 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.247771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.247874 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.247989 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.248882 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.249976 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.251485 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.252158 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.252287 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.256034 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.256041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.256046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.256088 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.260130 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.260973 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.260979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.260982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.261133 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.261594 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.261599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.261602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.261986 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.262133 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.262731 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.262971 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 21:06:28.264523 23305 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:28.265745 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.265209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57404)
0923 21:06:28.265498 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.265585 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 21:06:28.266903 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.266534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57416)
0923 21:06:28.266688 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.266786 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 21:06:28.267156 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.318548 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 1
I20260923 21:06:28.319820 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.319244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57432)
0923 21:06:28.319493 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.319613 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 21:06:28.320923 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.320539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57444)
0923 21:06:28.320701 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.320788 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 21:06:28.321238 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.350939 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 3
I20260923 21:06:28.352125 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.351626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57456)
0923 21:06:28.351857 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.351967 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 21:06:28.353264 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.352845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57458)
0923 21:06:28.353083 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.353155 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260923 21:06:28.353451 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.389608 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 2
I20260923 21:06:28.390838 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.390279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57470)
0923 21:06:28.390583 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.390680 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 21:06:28.391992 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.391623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57474)
0923 21:06:28.391782 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.391858 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 21:06:28.392198 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.496650 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 4
I20260923 21:06:28.497915 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.497357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57482)
0923 21:06:28.497619 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.497727 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 21:06:28.498425 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 3
I20260923 21:06:28.499094 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.498699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57492)
0923 21:06:28.498865 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.498948 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 21:06:28.499332 23379 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
I20260923 21:06:28.499524 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.499001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57498)
0923 21:06:28.499196 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.499297 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 21:06:28.500684 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.500218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57512)
0923 21:06:28.500420 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.500502 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 21:06:28.500957 23305 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33885: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33885: connect: Connection refused (error 111)
W20260923 21:06:28.504499 23387 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 21:06:28.504871 23387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 21:06:28.504980 23387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 21:06:28.505119 23387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 21:06:28.512830 23387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 21:06:28.512936 23387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 21:06:28.513023 23387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 21:06:28.513101 23387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 21:06:28.513206 23387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 21:06:28.523774 23387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38395,127.0.97.61:40277,127.0.97.60:33885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 19:58:29 UTC on ccycloud.quasar-janqnj.root.comops.site
build id 4668599
ASAN enabled
I20260923 21:06:28.525166 23387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 21:06:28.526643 23387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 21:06:28.539202 23392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 21:06:28.539182 23393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:28.540097 23387 server_base.cc:1034] running on GCE node
W20260923 21:06:28.540268 23395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 21:06:28.541115 23387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 21:06:28.542706 23387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 21:06:28.543990 23387 hybrid_clock.cc:648] HybridClock initialized: now 1790197588543904 us; error 56 us; skew 500 ppm
I20260923 21:06:28.544391 23387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 21:06:28.547919 23387 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260923 21:06:28.549346 23387 webserver.cc:460] Webserver started at http://127.0.97.60:34207/ using document root <none> and password file <none>
I20260923 21:06:28.550091 23387 fs_manager.cc:359] Metadata directory not provided
I20260923 21:06:28.550177 23387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 21:06:28.550523 23387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 21:06:28.553493 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data/instance:
uuid: "ce65e5da459849529c5b8c55af899660"
format_stamp: "Formatted at 2026-09-23 21:06:28 on dist-test-slave-8xn5"
server_key: "7edb5a03af0feb59716bdebcf628da5c"
server_key_iv: "266b7c9e212b28f7c4f4523b5acb0579"
server_key_version: "encryptionkey@0"
I20260923 21:06:28.554248 23387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/wal/instance:
uuid: "ce65e5da459849529c5b8c55af899660"
format_stamp: "Formatted at 2026-09-23 21:06:28 on dist-test-slave-8xn5"
server_key: "7edb5a03af0feb59716bdebcf628da5c"
server_key_iv: "266b7c9e212b28f7c4f4523b5acb0579"
server_key_version: "encryptionkey@0"
I20260923 21:06:28.559553 23387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 21:06:28.563138 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 21:06:28.564644 23387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 21:06:28.564872 23387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/wal
uuid: "ce65e5da459849529c5b8c55af899660"
format_stamp: "Formatted at 2026-09-23 21:06:28 on dist-test-slave-8xn5"
server_key: "7edb5a03af0feb59716bdebcf628da5c"
server_key_iv: "266b7c9e212b28f7c4f4523b5acb0579"
server_key_version: "encryptionkey@0"
I20260923 21:06:28.565145 23387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 21:06:28.588200 23387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 21:06:28.593300 23387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 21:06:28.593676 23387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 21:06:28.615860 23387 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33885
I20260923 21:06:28.615866 23453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33885 every 8 connection(s)
I20260923 21:06:28.617331 23387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/data/info.pb
I20260923 21:06:28.621112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu as pid 23387
I20260923 21:06:28.621304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDsUdkt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790197207951953-388-0/minicluster-data/master-2/wal/instance
I20260923 21:06:28.621411 23454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 21:06:28.621824 388 external_mini_cluster.cc:1428] Setting key 54f170298525c1735b41f496dc02f076
I20260923 21:06:28.631676 23454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260923 21:06:28.648139 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.623834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.0.97.60:33885)
0923 21:06:28.624776 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.624788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.624834 (+ 46us) server_negotiation.cc:407] Connection header received
0923 21:06:28.626515 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.626540 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.626981 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.627348 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.629194 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.630809 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.632704 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.633574 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.633735 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.639145 (+ 5410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.639155 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.639171 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.639218 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.642909 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.643910 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.643917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.643924 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.644114 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.644553 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.644558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.644561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.645318 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.645573 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.646066 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.646600 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":444,"threads_started":1}
I20260923 21:06:28.652794 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.634886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.0.97.62:38395)
0923 21:06:28.636377 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.636384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.636405 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:06:28.636550 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.636556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.636660 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.636787 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.638402 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.639375 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.641310 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.641991 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.642121 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.646147 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.646154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.646158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.646199 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.649509 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.650456 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.650460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.650462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.650604 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.651045 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.651049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.651052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.651421 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.651587 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.652149 (+ 562us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.652365 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":1180,"threads_started":1}
I20260923 21:06:28.652796 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.634759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38395 (local address 127.0.0.1:59330)
0923 21:06:28.635618 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.635668 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.635845 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.636902 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.636910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.637806 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.638189 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.638204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.639591 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.639595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.641145 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.641163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.642198 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.642203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.642389 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.643063 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.643087 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.645934 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.649661 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.649670 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.649677 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.650259 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.650721 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.650726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.650730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.650916 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.651579 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.651589 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.651799 (+ 210us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.652019 (+ 220us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.652368 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":368,"threads_started":1}
I20260923 21:06:28.654711 23454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:06:28.669926 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.651847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.0.97.62:38395)
0923 21:06:28.653049 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.653055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.653071 (+ 16us) server_negotiation.cc:407] Connection header received
0923 21:06:28.653129 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.653134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.653267 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.653374 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.655156 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.656098 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.657702 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.658220 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.658310 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.663358 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.663366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.663371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.663416 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.666517 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.667402 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.667408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.667412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.667559 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.668024 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.668029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.668032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.668430 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.668638 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.669166 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.669417 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":2814,"threads_started":1}
I20260923 21:06:28.669974 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.655608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40277 (local address 127.0.0.1:55692)
0923 21:06:28.655773 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.655794 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.655890 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.656266 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.656270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.656765 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.656963 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.656975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.658035 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.658039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.659405 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.659420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.660136 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.660145 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.660364 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.660929 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.660959 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.663406 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.667128 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.667135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.667141 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.667640 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.668046 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.668051 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.668054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.668211 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.668999 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.669002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.669200 (+ 198us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.669353 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.669654 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 21:06:28.670533 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.655646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.0.97.61:40277)
0923 21:06:28.655829 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.655835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.655856 (+ 21us) server_negotiation.cc:407] Connection header received
0923 21:06:28.656035 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.656041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.656123 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.656225 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.657109 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.657858 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.659567 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.659963 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.660066 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.663811 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.663818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.663821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.663866 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.666927 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.667749 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.667757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.667763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.667926 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.668376 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.668381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.668383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.668804 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.668989 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.669580 (+ 591us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.670024 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:06:28.671660 23454 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } has no permanent_uuid. Determining permanent_uuid...
I20260923 21:06:28.678212 23305 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 4
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260923 21:06:28.687101 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.672313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57536)
0923 21:06:28.672465 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.672482 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.672577 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.672932 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.672935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.673339 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.673470 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.673480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.674461 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.674469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.675336 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.675346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.676042 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.676048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.676225 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.676890 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.676913 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.679860 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.684149 (+ 4289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.684153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.684156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.684646 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.685108 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.685115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.685120 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.685289 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.685974 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.685980 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.686159 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.686371 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.686693 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 21:06:28.687999 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.672534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.97.60:33885)
0923 21:06:28.672685 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.672690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.672704 (+ 14us) server_negotiation.cc:407] Connection header received
0923 21:06:28.672759 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.672761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.672816 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.672915 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.673606 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.674290 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.675486 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.675877 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.676019 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.680096 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.680102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.680114 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.680159 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.683997 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.684781 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.684787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.684792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.684955 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.685414 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.685423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.685427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.685789 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.685964 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.687431 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.687687 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 21:06:28.690909 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.671859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.97.61:40277)
0923 21:06:28.672030 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.672037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.672057 (+ 20us) server_negotiation.cc:407] Connection header received
0923 21:06:28.672244 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.672250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.672330 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.672443 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.673778 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.674668 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.675959 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.676482 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.676638 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.682059 (+ 5421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.682068 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.682073 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.682121 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.687408 (+ 5287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.688409 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.688416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.688419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.688580 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.689030 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.689035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.689038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.689394 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.689695 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.690258 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.690539 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 21:06:28.694623 23454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: Bootstrap starting.
I20260923 21:06:28.695783 23308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.679140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57540)
0923 21:06:28.679865 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.679888 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.680029 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.680544 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.680550 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.681104 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.681337 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.681352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.683041 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.683049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.684434 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.684446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.685317 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.685325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.685546 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.686198 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.686216 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.688783 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.693040 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.693049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.693056 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.693569 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.693976 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.693981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.693985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.694168 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.694769 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.694773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.694967 (+ 194us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.695130 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.695441 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568}
I20260923 21:06:28.696324 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.679375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.0.97.60:33885)
0923 21:06:28.680175 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.680182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.680212 (+ 30us) server_negotiation.cc:407] Connection header received
0923 21:06:28.680272 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.680279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.680364 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.680502 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.681543 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.682875 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.684581 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.685147 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.685276 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.689044 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.689056 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.689061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.689090 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.692868 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.693713 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.693717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.693719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.693853 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.694283 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.694286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.694288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.694642 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.694793 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.695254 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.695485 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":699,"threads_started":1}
I20260923 21:06:28.699196 23454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:28.700367 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 21:06:28.700524 23454 log.cc:824] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:28.701822 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856: Bootstrap starting.
I20260923 21:06:28.703817 23454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: No bootstrap required, opened a new log
I20260923 21:06:28.706127 23305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:28.707661 23305 log.cc:824] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856: Log is configured to *not* fsync() on all Append() calls
I20260923 21:06:28.703514 23280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f8e1ca3559724c77be157bf0a8da6856"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59336
I20260923 21:06:28.711719 23454 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
I20260923 21:06:28.712106 23454 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce65e5da459849529c5b8c55af899660, State: Initialized, Role: FOLLOWER
I20260923 21:06:28.712584 23454 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
I20260923 21:06:28.713707 23305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856: No bootstrap required, opened a new log
I20260923 21:06:28.713610 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.714134 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:28.714953 23454 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:28.719578 23305 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
I20260923 21:06:28.719907 23305 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e1ca3559724c77be157bf0a8da6856, State: Initialized, Role: FOLLOWER
I20260923 21:06:28.720368 23305 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
I20260923 21:06:28.721335 23469 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.721740 23469 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:28.722647 23305 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:28.722877 23280 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 21:06:28.723101 23280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:28.727679 23280 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
I20260923 21:06:28.731842 23429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8e1ca3559724c77be157bf0a8da6856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ce65e5da459849529c5b8c55af899660"
I20260923 21:06:28.732424 23429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:28.736441 23429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e1ca3559724c77be157bf0a8da6856 in term 1.
I20260923 21:06:28.736533 23280 leader_election.cc:288] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [CANDIDATE]: Term 1 election: Requested vote from peers 577566b8817d48d692229f9970bfd1e6 (127.0.97.61:40277), ce65e5da459849529c5b8c55af899660 (127.0.97.60:33885)
I20260923 21:06:28.737188 23254 leader_election.cc:302] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce65e5da459849529c5b8c55af899660, f8e1ca3559724c77be157bf0a8da6856; no voters:
I20260923 21:06:28.737735 23469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 21:06:28.730391 23387 master_runner.cc:186] Error getting master registration for 127.0.97.62:38395: OK, instance_id { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" instance_seqno: 1790197587529482 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 21:06:28.744752 23387 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 21:06:28.748764 23481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 21:06:28.748939 23481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 21:06:28.749871 23469 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [term 1 LEADER]: Becoming Leader. State: Replica: f8e1ca3559724c77be157bf0a8da6856, State: Running, Role: LEADER
I20260923 21:06:28.754725 23469 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
W20260923 21:06:28.768242 23259 tablet.cc:2369] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:06:28.770005 23484 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8e1ca3559724c77be157bf0a8da6856. Latest consensus state: current_term: 1 leader_uuid: "f8e1ca3559724c77be157bf0a8da6856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.770519 23484 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:28.772670 23497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 21:06:28.780769 23238 master_runner.cc:186] Error getting master registration for 127.0.97.61:40277: OK, instance_id { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" instance_seqno: 1790197588065999 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 21:06:28.795814 23238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 21:06:28.798513 23379 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } attempt: 5
I20260923 21:06:28.799715 23497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 21:06:28.803222 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.754419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.0.97.61:40277)
0923 21:06:28.754626 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.754632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.768632 (+ 14000us) server_negotiation.cc:407] Connection header received
0923 21:06:28.773896 (+ 5264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.773903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.774028 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.774131 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.778725 (+ 4594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.779737 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.783023 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.783703 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.783805 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.787765 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.787774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.787781 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.787823 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.793474 (+ 5651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.794302 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.794310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.794313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.794514 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.794981 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.794989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.794994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.795396 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.795633 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.802555 (+ 6922us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.802832 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 21:06:28.803493 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.769614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.97.60:33885)
0923 21:06:28.769816 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.769822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.770977 (+ 1155us) server_negotiation.cc:407] Connection header received
0923 21:06:28.771277 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.771282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.771397 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.771484 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.774069 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.775329 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.777380 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.778002 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.778101 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.796103 (+ 18002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.796110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.796115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.796156 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.799873 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.800797 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.800802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.800805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.800970 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.801558 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.801563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.801566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.801949 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.802251 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.802891 (+ 640us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.803113 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":65}
Sep 23 21:06:28 dist-test-slave-8xn5 krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790197588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260923 21:06:28.816468 23429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [term 1 FOLLOWER]: Refusing update from remote peer f8e1ca3559724c77be157bf0a8da6856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:06:28.816597 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.754055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.97.62:38395)
0923 21:06:28.754246 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.754255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.759768 (+ 5513us) server_negotiation.cc:407] Connection header received
0923 21:06:28.760334 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.760343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.760455 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.760549 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.765755 (+ 5206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.767355 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.779199 (+ 11844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.779922 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.780046 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.786077 (+ 6031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.786085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.786092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.786138 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.795078 (+ 8940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.796714 (+ 1636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.796721 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.796725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.797052 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.797778 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.797810 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.797814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.806945 (+ 9131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.809509 (+ 2564us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.815400 (+ 5891us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.815876 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 21:06:28.818176 23484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:06:28.829286 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.799141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33885 (local address 127.0.0.1:57556)
0923 21:06:28.799569 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0923 21:06:28.799589 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 21:06:28.799691 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 21:06:28.800616 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 21:06:28.800622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 21:06:28.801141 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0923 21:06:28.801382 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.801395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.803090 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.803097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.804540 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 21:06:28.804557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.806274 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.806281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 21:06:28.806521 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.807225 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 21:06:28.807271 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 21:06:28.811943 (+ 4672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 21:06:28.822753 (+ 10810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.822760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.822764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.823553 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.824414 (+ 861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 21:06:28.824421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 21:06:28.824425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 21:06:28.824688 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 21:06:28.825832 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 21:06:28.825838 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 21:06:28.826092 (+ 254us) client_negotiation.cc:769] Sending connection context
0923 21:06:28.826377 (+ 285us) client_negotiation.cc:240] Negotiation successful
0923 21:06:28.827644 (+ 1267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260923 21:06:28.829762 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:28.799909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.0.97.60:33885)
0923 21:06:28.800107 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:28.800113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:28.800137 (+ 24us) server_negotiation.cc:407] Connection header received
0923 21:06:28.800190 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:28.800195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:28.800268 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:28.800360 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:28.801647 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.802863 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.805140 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:28.805860 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:28.806000 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:28.812258 (+ 6258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 21:06:28.812265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 21:06:28.812270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 21:06:28.812311 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 21:06:28.821969 (+ 9658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.823808 (+ 1839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.823814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.823818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.824003 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 21:06:28.824886 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 21:06:28.824891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 21:06:28.824894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 21:06:28.825650 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 21:06:28.826716 (+ 1066us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:28.827330 (+ 614us) server_negotiation.cc:299] Negotiation successful
0923 21:06:28.828235 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 21:06:28.835716 23379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6: Bootstrap starting.
I20260923 21:06:28.841143 23379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6: Neither blocks nor log segments found. Creating new log.
I20260923 21:06:28.843279 23379 log.cc:824] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6: Log is configured to *not* fsync() on all Append() calls
W20260923 21:06:28.846603 23408 tablet.cc:2369] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 21:06:28.850881 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8e1ca3559724c77be157bf0a8da6856. Latest consensus state: current_term: 1 leader_uuid: "f8e1ca3559724c77be157bf0a8da6856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.851227 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:28.853085 23379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6: No bootstrap required, opened a new log
I20260923 21:06:28.864555 23500 mvcc.cc:205] Tried to move back new op lower bound from 7332649323779756032 to 7332649323545284608. Current Snapshot: MvccSnapshot[applied={T|T < 7332649323779756032}]
I20260923 21:06:28.864499 23379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
I20260923 21:06:28.864920 23379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577566b8817d48d692229f9970bfd1e6, State: Initialized, Role: FOLLOWER
I20260923 21:06:28.865597 23379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } }
I20260923 21:06:28.880435 23469 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8e1ca3559724c77be157bf0a8da6856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.880944 23469 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:28.883311 23464 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8e1ca3559724c77be157bf0a8da6856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.883599 23464 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:28.888300 23497 catalog_manager.cc:1347] Generated new cluster ID: 8a1ea12af9674001876b594b97acdf76
I20260923 21:06:28.888414 23497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 21:06:28.899053 23505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.899726 23505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:28.900699 23379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 21:06:28.901142 23353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 21:06:28.904668 23354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8e1ca3559724c77be157bf0a8da6856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "577566b8817d48d692229f9970bfd1e6"
I20260923 21:06:28.909821 23353 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [term 1 FOLLOWER]: Refusing update from remote peer f8e1ca3559724c77be157bf0a8da6856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 21:06:28.913820 23354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e1ca3559724c77be157bf0a8da6856 in term 1.
I20260923 21:06:28.915263 23484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 21:06:28.933233 23497 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 21:06:28.935699 23497 catalog_manager.cc:1504] Loading token signing keys...
I20260923 21:06:28.945546 23505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8e1ca3559724c77be157bf0a8da6856. Latest consensus state: current_term: 1 leader_uuid: "f8e1ca3559724c77be157bf0a8da6856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.945926 23505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:28.951440 23469 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8e1ca3559724c77be157bf0a8da6856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.951781 23469 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856 [sys.catalog]: This master's current role is: LEADER
I20260923 21:06:28.953827 23514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8e1ca3559724c77be157bf0a8da6856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e1ca3559724c77be157bf0a8da6856" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38395 } } peers { permanent_uuid: "577566b8817d48d692229f9970bfd1e6" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40277 } } peers { permanent_uuid: "ce65e5da459849529c5b8c55af899660" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33885 } } }
I20260923 21:06:28.954429 23514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 21:06:28.966990 23497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8e1ca3559724c77be157bf0a8da6856: Generated new TSK 0
I20260923 21:06:28.968205 23497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 21:06:28.991142 23522 catalog_manager.cc:1259] Loaded cluster ID: 8a1ea12af9674001876b594b97acdf76
I20260923 21:06:28.991274 23522 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6: loading cluster ID for follower catalog manager: success
I20260923 21:06:29.011150 23522 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6: acquiring CA information for follower catalog manager: success
I20260923 21:06:29.015630 23522 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 577566b8817d48d692229f9970bfd1e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 21:06:29.130285 23459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.103351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.0.97.62:38395)
0923 21:06:29.103585 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.103594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.125170 (+ 21576us) server_negotiation.cc:407] Connection header received
0923 21:06:29.128602 (+ 3432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.128609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.128702 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.128775 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.130030 (+ 1255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59346: BlockingRecv error: recv got EOF from 127.0.0.1:59346 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 21:06:29.133533 23384 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.109658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55714 (local address 127.0.97.61:40277)
0923 21:06:29.109863 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.109870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.128475 (+ 18605us) server_negotiation.cc:407] Connection header received
0923 21:06:29.130859 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.130866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.130970 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.131034 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.133275 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55714: BlockingRecv error: recv got EOF from 127.0.0.1:55714 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 21:06:29.139652 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.125772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.0.97.60:33885)
0923 21:06:29.125955 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.125962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.132373 (+ 6411us) server_negotiation.cc:407] Connection header received
0923 21:06:29.132531 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.132537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.132620 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.132730 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.139373 (+ 6643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57570: BlockingRecv error: recv got EOF from 127.0.0.1:57570 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 21:06:29.197048 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.177284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.97.61:40277)
0923 21:06:29.177480 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.177486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.188202 (+ 10716us) server_negotiation.cc:407] Connection header received
0923 21:06:29.188595 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.188603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.188834 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.188999 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.189745 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.190983 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.193453 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.194155 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.194272 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.195137 (+ 865us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.195904 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.196044 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.196577 (+ 533us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.196677 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 21:06:29.257126 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.240342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.0.97.60:33885)
0923 21:06:29.240603 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.240614 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.255081 (+ 14467us) server_negotiation.cc:407] Connection header received
0923 21:06:29.255397 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.255404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.255542 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.255628 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.256809 (+ 1181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57572: BlockingRecv error: recv got EOF from 127.0.0.1:57572 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260923 21:06:29.307559 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.288091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.0.97.61:40277)
0923 21:06:29.288299 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.288306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.300088 (+ 11782us) server_negotiation.cc:407] Connection header received
0923 21:06:29.300654 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.300667 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.300768 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.300839 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.301671 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.302906 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.305014 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.305821 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.305925 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.306601 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.306772 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.306839 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.307088 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.307191 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 21:06:29.358776 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.345531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.0.97.60:33885)
0923 21:06:29.345767 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.345774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.356602 (+ 10828us) server_negotiation.cc:407] Connection header received
0923 21:06:29.357234 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.357240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.357346 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.357422 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.358537 (+ 1115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57588: BlockingRecv error: recv got EOF from 127.0.0.1:57588 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 21:06:29.390491 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.382751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.0.97.61:40277)
0923 21:06:29.382963 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.382970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.383207 (+ 237us) server_negotiation.cc:407] Connection header received
0923 21:06:29.383543 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.383549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.383653 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.383762 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.384307 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.385626 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.387903 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.388641 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.388779 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.389320 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.389489 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.389586 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.389806 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.389923 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 21:06:29.404621 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.402216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.0.97.60:33885)
0923 21:06:29.402427 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.402435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.403013 (+ 578us) server_negotiation.cc:407] Connection header received
0923 21:06:29.403205 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.403211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.403323 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.403449 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.404341 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57596: BlockingRecv error: recv got EOF from 127.0.0.1:57596 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 21:06:29.420467 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.412273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.0.97.61:40277)
0923 21:06:29.412545 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.412552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.413118 (+ 566us) server_negotiation.cc:407] Connection header received
0923 21:06:29.413328 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.413334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.413430 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.413547 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.414099 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.415434 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.418136 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.418955 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.419097 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.419623 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.419772 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.419878 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.420065 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.420160 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260923 21:06:29.432577 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.428443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.0.97.60:33885)
0923 21:06:29.428690 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.428697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.430918 (+ 2221us) server_negotiation.cc:407] Connection header received
0923 21:06:29.431148 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.431154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.431258 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.431375 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.432333 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57608: BlockingRecv error: recv got EOF from 127.0.0.1:57608 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 21:06:29.453706 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.445645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.0.97.61:40277)
0923 21:06:29.445858 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.445865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.446532 (+ 667us) server_negotiation.cc:407] Connection header received
0923 21:06:29.446755 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.446763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.446887 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.447023 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.447554 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.448846 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.451246 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.452037 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.452195 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.452751 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.452938 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.453050 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.453231 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.453337 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260923 21:06:29.463850 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.461331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.0.97.60:33885)
0923 21:06:29.461533 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.461540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.462142 (+ 602us) server_negotiation.cc:407] Connection header received
0923 21:06:29.462375 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.462382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.462492 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.462612 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.463570 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57624: BlockingRecv error: recv got EOF from 127.0.0.1:57624 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 21:06:29.490396 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.483333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.0.97.61:40277)
0923 21:06:29.483576 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.483583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.483978 (+ 395us) server_negotiation.cc:407] Connection header received
0923 21:06:29.484166 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.484173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.484281 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.484414 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.485064 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.486491 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.488121 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.488835 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.489063 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.489523 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.489679 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.489795 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.489947 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.490043 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260923 21:06:29.499337 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.497273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.60:33885)
0923 21:06:29.497496 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.497503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.497853 (+ 350us) server_negotiation.cc:407] Connection header received
0923 21:06:29.498021 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.498030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.498148 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.498290 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.499089 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57636: BlockingRecv error: recv got EOF from 127.0.0.1:57636 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 21:06:29.540959 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.534722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.0.97.61:40277)
0923 21:06:29.535051 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.535061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.535650 (+ 589us) server_negotiation.cc:407] Connection header received
0923 21:06:29.535895 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.535907 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.536023 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.536130 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.536590 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.537561 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.539130 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.539595 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.539733 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.540135 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.540258 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.540368 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.540518 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.540597 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
W20260923 21:06:29.552294 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.547393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.0.97.60:33885)
0923 21:06:29.547642 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.547651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.550869 (+ 3218us) server_negotiation.cc:407] Connection header received
0923 21:06:29.551101 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.551106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.551183 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.551293 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.552010 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57648: BlockingRecv error: recv got EOF from 127.0.0.1:57648 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 21:06:29.632258 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.625402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.97.61:40277)
0923 21:06:29.625590 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.625597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.626156 (+ 559us) server_negotiation.cc:407] Connection header received
0923 21:06:29.626377 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.626383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.626472 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.626571 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.627051 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.628133 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.630193 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.630923 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.631057 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.631520 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.631656 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.631742 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.631893 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.631975 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 21:06:29.640102 23457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 21:06:29.638098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57660 (local address 127.0.97.60:33885)
0923 21:06:29.638293 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.638300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.638691 (+ 391us) server_negotiation.cc:407] Connection header received
0923 21:06:29.638904 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.638913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.638983 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.639090 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 21:06:29.639885 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57660: BlockingRecv error: recv got EOF from 127.0.0.1:57660 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 21:06:29.751081 23481 catalog_manager.cc:1259] Loaded cluster ID: 8a1ea12af9674001876b594b97acdf76
I20260923 21:06:29.751183 23481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: loading cluster ID for follower catalog manager: success
I20260923 21:06:29.753693 23481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: acquiring CA information for follower catalog manager: success
I20260923 21:06:29.755635 23481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce65e5da459849529c5b8c55af899660: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 21:06:29.778118 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.772760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.0.97.61:40277)
0923 21:06:29.773007 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.773014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.773270 (+ 256us) server_negotiation.cc:407] Connection header received
0923 21:06:29.773532 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.773537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.773610 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.773699 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.774160 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.775044 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.776371 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.776799 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.776906 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.777333 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.777494 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.777568 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.777719 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.777786 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 21:06:29.791198 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.784146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.97.60:33885)
0923 21:06:29.784457 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.784464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.784866 (+ 402us) server_negotiation.cc:407] Connection header received
0923 21:06:29.785078 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.785106 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.785499 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.785620 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.786036 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.786930 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.788541 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.788980 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.789087 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.789752 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.790570 (+ 818us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.790704 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.790869 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.790946 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 21:06:29.806758 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.798424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.0.97.61:40277)
0923 21:06:29.798688 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.798697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.799423 (+ 726us) server_negotiation.cc:407] Connection header received
0923 21:06:29.799717 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.799723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.799828 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.799885 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.800695 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.802021 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.804360 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.805118 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.805264 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.805923 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.806069 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.806169 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.806370 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.806465 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 21:06:29.807179 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.798101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.0.97.62:38395)
0923 21:06:29.798707 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.798718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.799336 (+ 618us) server_negotiation.cc:407] Connection header received
0923 21:06:29.799834 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.799844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.800135 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.800348 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.801195 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.802471 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.804659 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.805086 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.805223 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.805895 (+ 672us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.806411 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.806515 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.806739 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.806840 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":615,"threads_started":1}
I20260923 21:06:29.808118 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 21:06:29.799533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.0.97.60:33885)
0923 21:06:29.799723 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 21:06:29.799729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 21:06:29.801248 (+ 1519us) server_negotiation.cc:407] Connection header received
0923 21:06:29.801525 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 21:06:29.801531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 21:06:29.801609 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 21:06:29.801710 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 21:06:29.802399 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.803223 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.805445 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 21:06:29.806126 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 21:06:29.806237 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 21:06:29.807216 (+ 979us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 21:06:29.807385 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 21:06:29.807482 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 21:06:29.807721 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 21:06:29.807793 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 21:06:29.813226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 23238
I20260923 21:06:29.814672 23300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:29.929981 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 23238
W20260923 21:06:29.951524 23406 connection.cc:535] server connection from 127.0.0.1:57540 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 21:06:29.952256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 23312
I20260923 21:06:29.953425 23374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:30.065126 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 23312
I20260923 21:06:30.087114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu pid 23387
I20260923 21:06:30.088258 23449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 21:06:30.196165 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDsUdkt/build/asan/bin/kudu with pid 23387
2026-09-23T21:06:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4610 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9304 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (382204 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST